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EXECUTIVE  SUMMARY 


Vision  is  the  dominant  information  gathering  sense.  It  is  a 
complex  resource  about  which  there  is  still  a  great  deal  to 
learn.  Human  beings  make  mistakes,  and  these  are  often  based  on 
the  assumption  that  they  have  seen  all  there  is  to  see.  They 
then  make  decisions  which  are  not  necessarily  grounded  on  the 
best  information  available.  In  most  daily  activities,  such 
errors  are  not  hazardous  and  can  be  recovered.  In  the  dynamic 
world  of  air  traffic  control  (ATC),  missing  a  piece  of  critical 
visual  information  can  cause  some  serious  consequences  leading  to 
an  operational  error.  One  of  the  most  frequent  comments  made  by 
controllers  who  have  had  an  error  is:  "I  didn't  see  it."  The 
nagging  question  which  follows  such  a  situation  is:  "Why  not"? 

While  every  situation  is  specific,  there  may  be  common  elements 
which  relate  to  how  controllers  use  their  vision  to  search  for 
information.  This  research  was  undertaken  to  learn  more  about 
controller  visual  scanning  in  a  radar  environment.  The  long  term 
objective  was  to  develop  a  better  understanding  of  controller 
vision  and,  by  doing  so,  help  controllers  avoid  errors  in  the 
future.  Previous  research  on  controller  scanning  has  been  very 
limited  and  has  focused  primarily  on  what  might  be  viewed  as 
obvious.  Radar  controllers  spend  the  majority  of  their  time 
looking  at  the  radar  and  proportionately  less  time  at  other 
sources  such  as  flight  progress  strips.  Prior  work  did  not 
attempt  to  measure  eye  movements  in  a  precise  manner  and  relate 
them  to  controller  performance.  This  was  the  goal  of  the  current 
project . 

There  was  no  working  definition  of  scanning  when  this  program 
began.  The  following  was  developed  to  fill  the  void. 

Scanning  refers  to  a  systematic  and  continuous  effort  to  acquire 
all  necessary  visual  information  in  order  to  build  and  maintain  a 
complete  awareness  of  activities  and  situations  which  may  effect 
the  controllers'  area  of  responsibility. 

During  this  project,  ten  current  air  traffic  controllers 
participated  in  a  simulation  experiment  which  was  based  on  the 
same  airspace  geometry  as  their  home  facility.  This  group 
included  six  full  performance  level  (FPL)  and  four  developmental 
controllers,  all  of  whom  volunteered  to  spend  a  week  at  the 
Federal  Aviation  Administration  (FAA)  Technical  Center.  They 
worked  alternatively  north  and  south  final  approaches  to  a 
simulated  level  V  Terminal  Radar  Approach  Control  Facility 
(TRACON).  During  each  simulation,  everything  that  occurred  in 
the  airspace  and  every  action  by  the  controllers  were  measured 
and  evaluated. 
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Eye  movements  were  measured  using  an  Applied  Systems  Laboratory 
oculometer.  This  employs  an  infrared  light  source  which  is 
mounted  on  a  helmet  worn  by  the  participant.  The  system  collects 
the  light  reflected  from  the  right  eye  and  computes  the  eye 
movements  by  measuring  the  relationship  of  the  center  of  the 
pupil  with  respect  to  the  center  of  the  corneal  reflection.  In 
lay  terms,  this  provides  information  on  the  number  and  duration 
of  fixations  along  with  the  size  of  the  movements  between 
fixations  known  as  saccades.  During  each  day's  activities,  a 
person's  eyes  are  constantly  moving,  and  three  to  five  fixations 
per  seconJ  is  common.  The  oculometer  that  was  used  is  an  older 
system  and,  unfortunately,  it  does  not  provide  detailed 
information  on  the  points  of  fixation  or  where  the  individual  is 
actually  looking.  However,  patterns  of  fixations  can  be  plotted. 


The  research  design  called  for  the  controllers  to  work  in  pairs, 
one  used  the  oculometer  and  one  who  did  not.  This  allowed  for  an 
evaluation  of  the  impact  of  the  oculometer  itself.  Other 
variables  of  interest  were  the  differences  in  performance  and 
scanning  behavior  between  the  two  groups  of  controllers,  FPL  and 
developmental,  and  whether  there  would  be  any  visual  differences 
produced  by  two  levels  of  work  load. 

When  asked  for  their  perceptions,  controllers  indicated  that  they 
found  the  occulometer  annoying,  but  generally  forgot  about  it 
when  they  were  busy.  Responses  on  a  post-run  questionnaire 
indicated  that  there  were  some  differences  of  perceptions  based 
on  controller  experience.  Developmental  controllers  were  more 
concerned  about  how  tired  they  felt,  while  the  FPL's  felt  there 
was  a  greater  impact  on  their  performance  of  metering  of  inbound 
aircraft . 

Automated  performance  data  were  collected  by  the  simulation 
itself.  This  was  supported  by  an  over  the  shoulder  performance 
and  workload  rating  accomplished  by  an  air  traffic  controller 
serving  as  a  member  of  the  research  team.  Correlations  between 
variables  indicated  that  developmental  controllers,  overall,  had 
slightly  more  between  sector  conflicts  than  did  FPL's.  However, 
this  varied  with  the  conditions  under  which  they  worked  as  will 
be  shown  below.  The  observer  rated  the  performance  of  the  FPL'S 
as  superior  to  that  of  the  developmentals .  This  may  have  been 
confounded  since  the  observer  knew  who  was  who.  However,  he  did 
his  best  to  be  objective.  Other  significant  correlations 
included  a  positive  relationship  between  task  load  and  standard 
conflicts  along  with  workload  ratings.  The  more  task  load,  the 
more  standard  conflicts  there  were,  and  higher  workload  was 
observed. 

Analyses  of  variance  were  computed  on  the  performance  data  and 
the  observer  ratings.  They  too  showed  a  number  of  significant 
relationships.  The  frequency  of  standard  conflicts  increased 
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with  task  load,  as  indicated  earlier  by  the  correlation  data. 

This  was  also  true  for  the  workload  ratings.  A  measure  of 
severity  for  between  sector  conflicts  (BAPI)  increased  as  a 
function  of  task  load.  These  findings  were  irrespective  of  skill 
level.  Two  performance  variables  produced  results  that  were 
complicated  by  the  skill  of  the  controllers  involved.  FPL 
controllers  had  fewer  between  sector  conflicts  when  wearing  the 
oculometer  than  when  they  were  not.  The  developmentals,  in 
contrast,  had  fewer  when  they  were  not  wearing  the  equipment. 

This  may  well  relate  to  levels  of  confidence  and  novelty  of  the 
situation.  One  might  speculate  that  the  FPL's  were  more  careful 
when  they  wore  the  equipment  and  the  developmentals  were  more 
anxious.  There  was  also  another  relationship  for  a  measure  of 
conflict  severity  for  between  sector  conflicts.  When  this  was 
analyzed  further,  the  only  significant  difference  occurred 
between  the  FPL's  and  developmentals.  When  neither  was  using  the 
oculometer,  the  FPL's  had  more  severe  conflicts.  Again,  this  may 
relate  to  the  level  of  confidence  each  group  felt  in  the 
simulation  environment. 

Eye  movement  data  were  collected  by  the  oculometer  and  consisted 
of  different  forms  of  two  basic  variables:  fixations  and  saccades 
along  with  plots  of  this  information.  Correlations  were  computed 
between  vision  and  performance  variables.  The  number  of 
keystrokes  (PKEY)  made  by  simulated  pilots  could  be  viewed  as  and 
indicator  of  controller  activity  level.  PKEY  was  inversely 
related  to  saccade  duration.  The  busier  the  controller  became, 
the  shorter  and  more  frequent  were  his  saccades.  Also,  saccade 
duration  was  inversely  related  to  observer  workload  ratings. 

This  meant  that  when  the  controller  was  receiving  higher  workload 
ratings,  his  saccades  were  of  shorter  duration. 

Results  indicated  a  multiple  correlation  of  R=  .857  when  all 
vision  variables  were  regressed  on  controller  skill  level.  This 
was  significant.  Three  vision  variables  made  relevant 
contributions  to  this  regression:  fixation  frequencies,  saccade 
durations,  and  pupil  means.  The  strongest  of  these  variables  and 
the  one  most  easy  to  see  in  the  table  of  vision  data  means  was 
fixation  frequency.  FPL  controllers  had  higher  fixation 
frequencies  than  did  the  developmentals.  They  scanned  the 
environment  more  and  dwelled  on  individual  points  on  the  radar 
less.  While  this  did  not  necessarily  lead  to  better  performance, 
it  may  mark  a  basic  difference  between  more  and  less  experienced 
controllers . 

Vision  data  on  selected  variables  were  analyzed  to  determine  if 
there  was  a  time  course  to  the  scanning  behavior  of  controllers. 
When  30-minute  simulations  were  broken  into  six  blocks  of  5 
minutes  and  analyzed,  it  was  apparent  that  there  were  some 
significant  changes  over  time.  In  the  first  5  to  10  minutes  both 
saccade  magnitude  and  duration  decreased,  then  stabilized  for  the 
remainder  of  the  period.  Fixation  durations  increased  during  the 
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first  5  minutes  of  the  simulation,  then  stabilized  for  the 
remainder  of  the  control  period.  These  changes  were  irrespective 
of  skill  level  or  task  load.  It  is  well  known  that  many  errors 
occur  during  the  beginning  of  a  shift  or  right  after  a  break. 
There  are  changes  in  visual  scanning  that  are  occurring  during 
the  same  time  period.  Qualitative  analyses  of  scan  plots 
produced  by  the  oculometer  itself  support  the  conclusion  that  it 
takes  some  time  for  the  scan  to  stabilize  and  that  for  the 
situation  described  in  the  simulation,  airspace  geometry  and 
traffic  flow  are  major  determiners  of  scan  pattern.  These  data 
suggest  that  the  controller  should  be  building  his/her  scan  for  a 
period  of  5  minutes  or  so  prior  to  taking  actual  control. 

At  the  conclusion  of  the  vision  and  simulation  portion  of  the 
project,  each  controller  completed  Cattell's  16  personality 
factor  questionnaire  (16  PF),  which  is  a  well  known  inventory  of 
personality  traits.  This  was  done  to  see  whether  there  were  any 
differences  between  the  two  groups  of  participants.  FPL's 
described  themselves  as  more  self-assured  and  imaginative  than 
did  the  developmentals  who  indicated  that  they  were  more 
apprehensive.  Relationships  with  other  measures  and  personality 
were  computed.  Participants,  primarily  FPL's  who  were  more 
concerned  about  the  impact  of  metering  on  their  performance 
indicated  on  the  16  PF  that  they  were  more  calm,  independent,  and 
unpretentious.  Overall,  personality  could  be  viewed  as  one  of 
the  resources  that  controllers  bring  with  them  to  the  work 
environment,  and  it  likely  has  an  impact  on  how  they  perceive 
their  environment. 

During  exit  interviews,  controllers  indicated  that,  on  the  whole, 
they  were  comfortable  with  the  simulation  and  did  not  see  any 
major  impact  of  the  oculometer  itself.  Almost  all  felt  that 
their  scanning  methods  were  situation  dependent.  What  they 
considered  important  included;  traffic  volume  and  flow  aircraft 
speeds,  weather,  flow  control,  and  critical  points.  Verbalized 
strategies  agreed  with  what  was  seen  on  the  scan  plots.  When 
asked  about  separation  strategy  preferences,  the  controlleis 
confirmed  previous  studies  in  that  other  things  being  equal,  they 
preferred  vertical  separation  techniques. 
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INTRODUCTION 


In  the  world  of  human  perception,  vision  is  generally  accepted 
as  the  dominant  means  of  gathering  information.  It  is, 
however,  infinitely  more  complex  than  many  are  willing  to 
believe.  Yet  people,  highly  trained  and  competent 
professionals,  make  mistakes.  They  make  judgments  based  on 
the  belief  that  they  have  absorbed  all  there  is  to  see,  when 
there  was  data  available  right  in  front  of  them  that  they  did 
not  see.  There  is  no  dictum  that  quite  covers  this.  If  there 
was,  it  might  read:  "If  you  don't  look  at  it,  you  can't  see 
it.  " 

In  air  traffic  control  (ATC),  an  operational  error  occurs  when 
there  is  a  loss  of  separation  between  two  or  more  aircraft 
which  resulted  from  something  that  the  air  traffic  controller 
did  or  failed  to  do.  Fortunately,  the  vast  majority  of  these 
errors  are  technical  violations  of  the  rules  for  minimum 
separation  but  do  not  lead  to  catastrophic  results.  However, 
all  errors  must  be  taken  very  seriously,  and  controllers 
making  them  are  encouraged  to  do  better.  During  the  period 
from  September  1990  to  August  1991  there  were  749  operational 
errors  recorded  in  the  ATC  work  force  (FAA,  1991).  This 
represented  a  decline  of  14  percent  from  the  same  period  of 
the  previous  year. 

No  supervisor,  leader,  or  researcher  can  judge  what  another 
human  being  sees  or  does  not  see.  They  can  only  infer  based 
on  what  the  individual  says  that  he  sees  and  what  he 
subsequently  does  about  it.  In  a  highly  complex  person- 
machine  system  like  ATC,  defining  and  avoiding  human  error  is 
a  major  concern.  One  of  the  most  frequent  statements  made  by 
personnel  who  have  made  a  mistake  is:  ”I  didn't  see  it."  In 
1987,  a  Federal  Aviation  Administration  (FAA)  administrator's 
task  force  on  controller  error  reduction  identified  two  key 
areas  of  concern:  memory  lapses  and  visual  scanning.  In  an 
effort  to  understand  why  and  to  subsequently  reduce  human 
errors  in  ATC,  the  FAA  initiated  a  program  in  Controller 
Information  Scanning. 

PROGRAM  BACKGROUND. 


The  ATC  System  has  evolved  over  the  years  in  response  to  user 
needs  and  available  technology.  Historically,  the  ATC  process 
has  been  very  human  centered  and  dependent  on  the  ability  of 
air  traffic  controllers  (Thomas,  1985).  A  great  deal  of  what 
controllers  do  involves  information  acquisition  and  processing 
(Kirchner  and  Laurig,  1971;  Sperundio,  1971).  Everyone  has 
limits  in  terms  of  the  amount  of  information  they  can 
reasonably  handle  (Finkelman  and  Kirchner,  1980).  With  the 
technology  of  the  1980 's,  controllers  must  attend  to  a  wide 
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range  of  detail.  The  future  holds  out  the  hope  of  machine 
assistance  for  the  management  of  information.  It  remains 
unclear  what  the  impact  will  be  on  human  operators  (Jenny  and 
Ratner,  1974).  Controller  acquisition  of  knowledge  and  how  it 
will  be  used  are  complex  questions  which  bear  further  study 
(Spettell  and  Liebert,  1986;  Warm  and  Dember,  1986). 

Regardless  of  when  or  what  promises  technology  holds  for  the 
future,  the  FAA  still  must  deal  with  the  present  and  the 
everyday  potential  for  human  error.  Controller  vision  may  be 
one  area  in  which  increased  knowledge  can  help  reduce 
operational  errors. 

"Man  reveals  many  of  his  secrets  in  the  pattern  of  his  eye 
movements,  a  fact  appreciated  by  oriental  merchants,  poets  and 
policemen  at  least  as  long  as  it  was  by  psychologists" 

(Alpern,  1971,  p  369).  Our  ability  to  discriminate  detail, 
often  referred  to  as  our  visual  acuity,  drops  off  rapidly  as 
light  impinges  further  away  from  the  eyes'  point  of  clearest 
vision  -  the  fovea.  The  eyes  are  practically  in  constant 
motion  driven  by  six  muscle  groups  which  are  among  the  fastest 
in  the  human  body.  We  have  the  illusion  of  a  stable  visual 
field  despite  the  constant  eye  movements  and  despite  the  fact 
that  when  the  eyes  are  moving  they  are  not  taking  in  any 
information.  Our  impression  of  spatial  solidity  is  created 
by  our  central  nervous  system,  and  the  movement  of  the  eye 
muscles  is  coordinated  by  a  switching  circuit  in  the  brain 
stem.  Actually,  our  eyes  are  not  in  constant  motion.  If  that 
were  true,  we  would  be  unlikely  to  see  anything  at  all. 
Saccadic  movement  only  occupies  about  ten  percent  of  the  total 
viewing  time  (Norton  and  Stark,  1971).  The  majority  of  time 
during  our  waking  hours  the  eyes  are  stopped  or  fixated  on 
objects  and  events  in  the  surroundings.  This  is  when  they  are 
acquiring  information  to  reduce  our  level  of  uncertainty. 

A  literature  review  was  conducted  in  1989  to  identify  the 
amount  and  nature  of  whatever  had  been  done  on  eye  movement 
research  in  aviation  (Stein,  1989).  It  was  clear  that 
considerable  work  had  been  accomplished  in  terms  of  so  called 
vigilance  studies,  which  generally  involve  basic  signal 
detection  paradigms.  For  example,  Thackray,  Touchstone,  and 
Bailey  (1978)  reported  studying  the  vigilance  of  men  and  women 
using  e  simulated  radar  task.  However,  they  were  studying 
detection  latencies  as  a  function  of  time  on  duty  rather  than 
measuring  details  of  actual  eye  movements.  The  review  also 
indicated  that  when  researchers  chose  to  evaluate  eye  movement 
data,  there  were  many  alternative  ways  of  doing  it,  ranging  in 
complexity  from  the  informal  observations  of  another  human 
serving  as  observer  to  a  variety  of  hi-tech  systems,  each  of 
which  had  assets  and  limitations.  Methods  varied  considerably 
in  terms  of  the  level  of  their  intrusiveness  on  the  operator. 
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It  was  apparent  that  while  a  good  deal  of  research  had  been 
conducted  in  studying  the  behavior  of  pilots  in  the  cockpit 
and  in  numerous  laboratory  vigilance  studies,  precious  little 
had  been  accomplished  in  the  ATC  environment.  There  have  been 
a  few  limited  studies  of  controller  eye  movements.  Wallis  and 
Samuel  (1961)  used  a  technique  called  electro-oculography, 
which  literally  measures  muscle  movements  around  the  eyes,  but 
can  generate  little  more  than  patterns  of  movement.  Karston, 
Goldberg,  Rood,  and  Sultzer  (1975)  evaluated  the  potential 
usefulness  of  an  early  oculometer  to  measure  visual  behavior. 
They  were  able  to  demonstrate  what  was  conventional  wisdom 
then  and  today.  Controllers  spend  the  majority  of  their  time 
looking  at  the  radar  display  and  proportionately  less  time 
searching  other  data  sources.  Thackray  and  Touchstone  (1980) 
used  electro-oculography  to  evaluate  the  impact  of  radar  sweep 
lines  on  scan  patterns  and  David  (1985)  used  a  rather 
cumbersome  video-based  system  to  examine  the  impact  of  color 
on  radar  and  other  data  displays. 

What  has  become  apparent  is  that  while  there  is  some 
literature  on  controller  eye  movements,  it  is  very  limited; 
and  this  is,  in  essence,  an  unexplored  area  which  may  well 
serve  as  a  window  into  air  traffic  controller  information 
processing.  Virtually  nothing  is  known  about  how  controllers 
scan  for  information.  Anything  which  is  learned  as  a  result 
of  this  study  will  add  to  the  body  of  knowledge  and  eventually 
help  the  research  community  to  assist  controllers  in  doing 
their  jobs  more  effectively. 

A  meeting  was  convened  in  June  1987  by  Air  Traffic 
Requirements  (ATR)  to  discuss  the  nature  of  scanning  and  its 
impact  on  operations  and  training.  It  was  noted  that  while 
automation  had  increased,  the  number  of  aircraft  that  a  single 
controller  could  work  had  not  increased  appreciably,  and 
controllers  continued  to  make  the  same  sort  of  mistakes. 

These  were  often  attributed  to  a  failure  to  perceive  critical 
information.  However,  despite  the  ease  with  which  people  used 
the  concept  of  "scanning"  and  the  fact  that  whatever  it  was, 
it  "has  been  done  forever,"  no  one  had  ever  really  defined 
scanning  in  an  ATC  context. 

Definition;  there  is  currently  no  one  generally  acceptable 
definition  of  the  concept  of  scanning.  It  grew  out  of  what 
appears  to  have  been  an  attempt  to  develop  an  operational 
label  for  the  problems  seen  in  the  applied  ATC  environment. 

The  following  is  offered  as  a  tentative  working  definition; 

"Scanning  refers  to  a  systematic  and  continuous  effort  to 
acquire  all  necessary  visual  information  in  order  to  build  and 
maintain  a  complete  awareness  of  activities  and  situations 
which  may  affect  the  controllers  area  of  responsibility." 


3 


This  definition  implies  the  ideal  of  a  motivated  operator  who 
is  reaching  out  for  all  necessary  information  and  is  managing 
the  data  flow  efficiently.  Anything  which  inhibits  this 
process  may  well  lead  to  an  operational  error. 

METHOD 


PARTICIPANTS. 


Personnel  involved  as  participants  in  this  study  were 
qualified  ATC  specialists  from  an  operational  Terminal  Radar 
Approach  Control  Facility  (TRACON)  in  a  busy  metropolitan 
area.  They  volunteered  to  come  to  the  Technical  Center  for 
several  days.  All  controllers  were  briefed  concerning  their 
rights  to  informed  consent  and  anonymity.  Participants  were 
current  in  approach  control  procedures  and  had  worked  active 
traffic  in  the  past  3  months.  Controllers  were  selected  from 
volunteer  applicants  based  on  their  level  of  experience  in  ATC 
and  whether  they  met  basic  criteria  for  study  suitability. 

Experience  was  supposed  to  be  from  two  levels.  Developmental 
controllers  who  have  completed  the  radar  school  and  have  been 
on  the  boards  for  1  year  or  less  were  originally  specified  as 
the  journeyman  sample.  However,  since  we  were  recruiting  from 
a  very  busy  TRACON,  what  actually  occurred  was  that  the 
journeymen  controllers  were  also  fairly  experienced,  but  were 
new  to  the  facility  and  had  to  recertify  after  transferring 
from  lower  level  operations  at  other  TRACON' s.  The  four 
developmental  controllers  had  an  average  of  8.35  years  in  ATC 
(range  7.25  to  10.5)  with  only  a  mean  facility  experience  of 
1.1  years.  The  six  Full  Performance  Level  (FPL)  controllers 
had  an  average  of  10.68  years  of  experience  (range  7  to  13.75) 
with  a  mean  time  in  the  facility  of  4.25  years.  The  primary 
difference  between  the  two  groups  was  their  experience  in  the 
facility  in  which  they  were  now  working.  Everyone  reported  on 
their  entrance  questionnaire  that  their  vision  was  good,  they 
were  in  good  general  health,  had  freely  volunteered  to 
participate,  and  that  they  were  motivated  to  participate. 

They  were  also  asked  to  report  on  a  10-point  scale  the  current 
level  of  stress  in  their  lives.  While  there  was  no 
appreciable  difference  between  the  two  groups  (mean  FPL  3.5 
and  mean  developmental  4.5),  there  was  a  considerable  range  of 
responses  from  1  (no  stress)  to  8  (signifying  fairly  high 
stress ) . 

Participants  had  to  be  physically  and  mentally  qualified  to 
perform  active  ATC  operations.  Due  to  the  relative  shortage 
of  full  performance  level  controllers,  participants 
volunteered  on  an  as  and  where  available  basis.  No  pretext  of 
systematic  sampling  is  made.  However,  participants  were 
selected  by  the  TRACON  from  a  pool  of  volunteers.  The 
Technical  Center  requested  that  selections  be  made  to  cover 
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the  normal  range  of  performance  abilities  to  help  enhance 
general izability. 

QUALIFICATIONS. 

This  was  a  small  sample  study  using  available  volunteers  from 
one  major  urban  tower/TRACON  facility.  While  every  effort  was 
made  to  accomplish  as  much  as  is  scientifically  possible  with 
the  limited  number  of  controllers  available,  any  results 
should  be  viewed  as  indicative  rather  than  conclusive. 
Subsequent  decisions  concerning  changes  should  be  done  using 
all  information  available  including  expert  judgement,  possible 
replications  of  this  study,  and  old  fashioned  common  sense. 

SIMULATION  FACILITY. 

This  study  was  accomplished  using  the  National  Airspace  System 
Simulation  Support  Facility  (NSSF),  which  is  an  ATC  simulator 
at  the  FAA  Technical  Center,  Atlantic  City  International 
Airport,  New  Jersey.  The  NSSF  is  a  general  purpose  ATC 
simulator  designed  to  provide  a  realistic  test  bed  for 
developing,  testing,  and  evaluating  advanced  ATC  concepts, 
airspace  management  plans,  and  procedures.  The  simulator 
consists  of  three  subsystems:  the  Controller  Laboratory,  the 
Simulator  Pilot  Complex,  and  the  Central  Computer  Facility. 

The  Controller  Laboratory  is  a  simulated  en  route  or  terminal 
control  room,  which  includes  eight  radar  displays  and  the 
associated  keyboard  entry  and  communication  equipment.  The 
laboratory  is  configured  so  that  the  participant  controllers 
can  function  in  a  manner  nearly  identical  to  the  way  they  do 
in  the  field.  Controller-to-controller ,  controller-to-pilot 
(simulator  operator),  and  pilot-to-controller  communications 
are  available  and  were  used  in  this  simulation.  The 
controller  portion  or  subsystem  provides  the  sights  and  sounds 
of  the  ATC  control  room.  While  it  is  not  a  perfect  copy  of 
the  radar  room  of  an  approach  control  (stimulus  fidelity),  it 
does  provide  fairly  realistic  opportunities  for  controller 
reactions  to  a  variety  of  real  world  situations  (response 
fidelity) . 

The  second  subsystem  of  the  NSSF  involves  people  who  serve  as 
the  "pilots"  of  the  aircraft  under  control.  These  Simpilots 
are  in  voice  contact  with  the  controller  and  respond  to  his 
directions.  They  fly  their  computer  generated  aircraft  from  a 
keyboard  in  an  adjacent  room.  One  Simpilot  controls  the 
flight  of  up  to  ten  aircraft. 

The  final  subsystem  is  the  computer,  which  serves  as  both  a 
target  generator  and  as  the  collector  of  all  systems 
information.  This  computer,  a  Gould  SEL,  samples  the 
simulated  airspace  every  second  and  records  all  aircraft 
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information  to  be  described  in  more  detail  in  a  subsequent 
paragraph. 

The  operation  of  the  simulation  facility  was  the 
responsibility  of  the  test  director.  He  coordinated  with  the 
technicians,  simulator  operators,  computer  operators,  and 
other  personnel  and  organizations  associated  with  the  test 
effort . 

The  standard  NSSF  data  reduction  program  was  used  to  provide 
the  following  data  elements  (table  1); 

TABLE  1.  DATA  MEASURES 


LCNF . Longitudinal  Conflicts 

LAPI . Longitudinal  API* 

MLAPI . Median  Longitudinal  API 

DLCNF . ...Duration  Longitudinal  Conflicts 

SCNF . Standard  Conflicts 

SAPI . Standard  Conflict  API 

SAPI . Median  Standard  API 

BCNF . Between  Sector  Conflicts 

BAPI . Between  Sector  API 

MBAPI . Median  Between  Sector  API 

PKEY . Total  Pilot  Key  Strikes 

LAND . Arrival  Landings 


*  To  be  defined  later 
TEMPORAL  DATA. 


The  primary  purpose  of  this  effort  was  to  evaluate  the  impact 
of  the  independent  variables  on  the  eye  movements  and 
performance  of  the  participant  controllers.  Temporal  data 
will  consist  of  controller  fixation  frequencies  and  durations 
collected  with  the  use  of  the  Applied  Systems  Laboratory  (ASL) 
oculometer.  The  concept  of  the  oculometer  and  its  operation 
are  presented  below. 

EYE  MOVEMENT  DATA  COLLECTION. 


The  system  employs  an  infrared  (IR)  light  source  which  is 
mounted  on  a  helmet  worn  by  the  participant.  The  optical 
system  collects  the  light  reflected  from  the  eye.  A 
participant's  eye  movement  with  respect  to  the  head  and, 
subsequently,  with  respect  to  the  point  of  fixation,  is 
computed  by  the  measurement  of  the  center  of  the  pupil  and  its 
relationship  to  the  center  of  the  corneal  reflection.  When 
the  entire  head  moves,  the  center  of  the  pupil  and  the  center 
of  corneal  reflection  move  together.  When  the  eyes  move  in 
the  head,  the  relationship  of  the  pupil  and  the  corneal 
reflection  changes  proportionally  based  on  the  degree  of  eye 
movement . 
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Once  the  system  is  set  up  and  the  participant  is  in  place,  the 
operator  of  the  oculometer  manually  acquires  the  center  of 
pupil,  the  operator  makes  adjustments  which  set  thresholds  for 
the  pupil  discriminator  and  the  corneal  reflection 
discriminator  so  that  the  detections  by  the  computer 
algorithms  are  reliable.  The  operator  then  performs  a  9- 
point  calibration  of  the  individuals  eye  movements  while 
he/she  maintains  head  position.  The  optics  or  optical  head 
coupled  to  the  computer  tracks  the  participants  eyes,  and  the 
system  allows  limited  head  movement  as  long  as  the  eye  image 
remains  centered  in  the  field  of  view.  The  system  is  coupled 
with  a  video  camera  and  monitor  so  that  the  operator  can  view 
the  participant's  eye  along  with  the  point  of  gaze 
superimposed  as  crosshairs  on  a  video  image  of  the  screen. 

The  oculometer  is  quite  typical  of  this  level  of  eye 
measurement  instruments.  It  records  pupil  diameter  and  works 
best  when  the  pupil  exceeds  3  millimeters.  Its  accuracy  is 
approximately  +/-  1  degree.  It  allows  about  1  cubic  foot  of 
head  movement.  If  the  participant  exceeds  that,  then  pupil 
acquisition  is  lost  and  has  to  be  reacquired.  Loss  of 
tracking  can  also  occur  by  prolonged  eye  blink,  coughing, 
sneezing,  or  anything  which  might  mask  the  pupil  or  corneal 
reflection.  One  positive  aspect  of  this  particular  system  was 
that  when  tracking  was  lost,  it  most  frequently  reacquired  the 
pupil  on  its  own.  The  other  manner  of  reacquisition  would 
involve  the  technician  doing  the  job  manually  with  a  joy  stick 
controller.  The  system  collects  calibrated  X  and  Y 
coordinates  of  fixation  points  on  the  stimulus  scene  at  a 
sampling  rate  of  60  per  second.  The  sampling  rate  is  limited 
by  the  scan  rate  of  the  television  equipment.  While  this 
system  is  not  perfect,  it  does  allow  more  flexibility  and  less 
obtrusiveness  than  those  methods  requiring  complete  head 
restrictions.  It  will  collect  fixation  position  and  duration. 
The  system  can  handle  large  amounts  of  digital  data  with 
minimal  manual  input. 

RESEARCH  DESIGN. 


This  experiment  included  three  independent  variables  each  of 
which  had  two  levels.  They  were  as  follows:  (1)  controller 
experience,  high  and  low,  (2)  system  task  load,  high  and  low, 
and  (3)  use  of  oculometer,  yes/no.  The  purpose  of  the  third 
variable  was  to  determine  if  the  use  of  the  oculometer  itself 
had  an  impact  on  the  controllers  performance  as  measured  by 
the  data  generated  by  the  simulation  itself.  The  overall 
design  is  depicted  in  figure  1.  The  design  will  employ 
repeated  measures  on  the  task  load,  oculometer  variables,  and 
independent  measures  on  the  subject  variable  of  experience. 
Participants  were  randomly  assigned  to  a  preselected 
administrative  order  of  those  conditions  on  which  repeated 
measures  were  collected.  The  administrative  orders  were  used 
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FIGURE  1.  RESEARCH  DESIGN 
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to  try  and  counterbalance  the  potential  order  effects  of  task 
load,  high  versus  low,  and  the  presence  or  absence  of  the 
oculometer.  However,  due  to  the  final  sample  size,  complete 
balancing  was  not  possible. 

PARTICIPANT  ACTIVITY. 

This  study  originally  required  18  active  controllers  from  an 
operating  TRACON.  All  participants  were  volunteers  who  had 
freely  agreed  to  travel  to  Atlantic  City  and  work  in  the  NSSF 
for  approximately  3  days.  All  personnel  came  from  the  same 
operational  facility  which  we  simulated  in  the  NSSF.  This 
greatly  reduced  the  time  required  to  familiarize  them  with  the 
simulation  and  increased  the  throughput  of  testing.  In  all 
cases,  the  participants  were  guaranteed  both  anonymity  and 
confidentiality.  No  record  of  their  individual  performance 
was  kept  by  name  or  other  personal  identifier. 

PROCEDURE . 


When  the  controllers  arrived  they  were  briefed  on  the 
background  of  the  simulation  effort:  how  the  simulation  will 
be  conducted  and  what  was  expected  of  them.  Essentially,  the 
controllers  were  advised  to  function  as  they  normally  do. 

They  were  also  advised  that  they  would  be  given  a 
questionnaire  to  be  completed  after  every  test  run  and  that 
there  would  be  a  debriefing  at  the  conclusion  of  their  test 
participation.  The  purpose  of  the  debriefing  was  to  solicit 
feedback  from  the  controllers  on  the  overall  simulation  and 
any  areas  which  could  be  improved. 

The  progression  of  a  participant  controller  through  the 
experiment  is  described  graphically  in  figure  2.  After  an 
initial  welcome  and  description  of  project  goals  to  include  an 
informed  consent  briefing,  the  controllers  were  asked  to 
complete  a  brief  questionnaire  describing  their  background  in 
ATC  and  current  motivation  for  this  project.  Once  entry 
processing  was  completed,  a  period  of  training  and 
familiarization  began  so  that  results  in  the  experiment  would 
not  be  confounded  by  a  learning  curve  as  the  controller  tried 
to  understand  the  simulator  and  the  airspace  as  presented. 

This  training  was  based  on  the  Instructional  Systems  Design 
(ISD)  model  which  called  for  the  periodic  evaluation  of 
progress  and  for  feedback  of  the  results  of  that  evaluation  to 
both  the  trainee  and  to  the  training  system.  The  evaluation 
was  primarily  based  on  expert  judgement  of  in-house 
observer /evaluators .  The  training  objectives  included  tasks, 
conditions,  and  standards  as  described  the  "Training- 
Familiarization  Plan”  which  is  attached  as  appendix  A  to  this 
report.  At  the  end  of  the  first  few  hours  of  training,  a 
decision  point  was  reached  as  to  whether  or  not  the 
participant  was  ready  or  should  have  more  training.  The  ten 
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FIGURE  2.  EXPERIMENTAL  PROGRESSION 


participants  who  came  to  the  Technical  Center  all  checked  out 
on  the  simulation  in  the  2-hour  block  of  training  and 
familiarization,  and  no  additional  training  was  deemed 
necessary . 

Prior  to  the  beginning  of  data  collection,  each  participant 
was  assigned  to  one  of  the  preselected  administrative  orders 
of  the  different  combinations  of  the  independent  variables. 

All  data  collection  was  accomplished  using  an  arbitrary  letter 
code  preassigned  to  each  participant.  No  names  were  recorded 
on  any  forms  and  the  list  of  names  by  codes  maintained 
exclusively  by  the  experimenter  was  destroyed  at  the 
conclusion  of  the  experiment.  This  was  to  protect  the  privacy 
of  the  participants  and  to  encourage  their  openness  and 
honesty  when  completing  questionnaires  and  interviews. 

Each  controller  participant  served  in  four  simulation  runs 
using  an  ATC  environment  that  was  based  on  his  home  facility. 
Two  of  these  runs  were  at  low  task  load  and  two  were  at  high 
load.  Controllers  participated  in  pairs,  and  during  each  run 
one  wore  the  oculometer  and  one  did  not.  So  each  controller 
had  one  oculometer  run  at  each  task  load. 

A  typical  data  collection  run  proceeded  as  follows:  The 
experimenter  informed  the  simulation  manager  prior  to  the  run 
of  the  conditions  under  which  the  data  will  be  collected.  The 
simulation  was  set  up  accordingly.  If  the  oculometer  was  to 
be  used,  the  participant  sat  down  and  received  the  calibration 
procedure.  Once  this  was  completed,  the  participant  received 
an  airspace  briefing  from  and  air  traffic  controller  familiar 
with  the  simulation  and  the  airspace  in  use  and  was  given  a 
few  minutes  to  evaluate  and  plan  as  he/she  saw  fit.  The 
participant  then  took  control  of  the  airspace  in  the 
designated  role.  Each  data  run  was  1/2  hour  and  involved  free 
play  simulation  in  which  the  participants  made  all  the 
decisions  normally  made  by  an  individual  in  his  position. 

Data  collection  occurred  both  during  and  after  each  simulation 
run.  During  the  run,  it  consisted  of  both  manual  and 
automated  methods.  The  manual  system  was  based  on  the 
continuous  observation  of  an  observer /evaluator  who  made 
entries  every  15  minutes  on  an  evaluation  form,  a  copy  of 
which  can  be  found  in  appendix  B.  The  automated  system 
involved  the  continuous  sampling  by  the  simulation  itself  of 
systems  variables  (see  table  1  for  a  sampling  of  these)  which 
include  aircraft  status,  changes  in  status,  separation  between 
aircraft  pairs,  and  participant  controller  actions.  The 
simulation  system  can  provide  these  data  in  raw  form  or  with  a 
considerable  amount  of  processing  to  include  cumulation  over 
time  intervals.  The  oculometer  subsystem  served  as  an 
additional  automated  data  collector  sampling  eye  movements  at 
a  rate  of  60  frames  per  second  and  recording  the  data  in  the 
ASL  format.  After  each  data  run,  the  participant  was  asked  to 
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complete  a  questionnaire  (shown  in  appendix  B)  designed  to 
gauge  their  assessment  of  how  hard  they  had  to  work  on  that 
run  and  how  they  felt  that  they  performed.  They  were  also 
asked  for  whatever  strategies  that  they  established  for  the 
run  and  for  their  subjective  impressions  of  the  impact  of  the 
oculometer  if  applicable  for  that  run. 

At  the  conclusion  of  all  the  data  runs,  each  controller 
participant  was  given  an  exit  interview  (see  appendix  B)  and 
asked  to  complete  a  personality  inventory  called  the  "Sixteen 
Personality  Factor  Questionnaire"  (Cattell,  Eber,  and 
Tatsuoka,  1970).  This  instrument  is  used  as  part  of  the 
entrance  testing  for  new  air  traffic  controllers,  and  was 
employed  to  determine  whether  there  would  be  any  differences 
between  the  two  small  groups  of  controllers  who  participated 
based  on  their  status  as  developmentals  or  FPLs. 

RESULTS 


CONTROLLER  QUESTIONNAIRE. 

At  the  end  of  each  1/2  hour  of  simulation,  controllers  were 
asked  to  complete  a  questionnaire  which  basically  described 
their  experience  in  the  simulation.  They  were  asked  to 
evaluate  their  workload,  self-assess  their  performance,  and 
provide  information  on  the  following  scales;  stress,  level  of 
interference  by  the  oculometer,  traffic  volume  and 
composition,  impact  of  runway  layout  and  aircraft  at  the  edges 
of  the  display,  metering  of  in-bound  aircraft,  and  personal 
fatigue.  The  mean  participant  responses  to  these  questions 
are  presented  in  table  2. 

A  visual  inspection  of  this  table  indicates  the  possibility 
that  participants  found  the  presence  of  the  oculometer 
annoying.  Controllers  reported  a  moderate  degree  of 
interference  from  the  oculometer.  This  was  not  at  all 
surprising  given  the  weight  of  the  helmet  and  the  restrictions 
that  it  engendered.  It  was  also  apparent  that  FPL  controllers 
were  more  confident  of  their  level  of  performance  with  and 
without  the  oculometer  than  were  the  developmentals.  No  one 
indicated  that  the  experience  overall  was  particularly 
stressful,  especially  after  they  adapted  to  the  presence  of 
the  oculometer  and  the  simulation  itself.  The  developmentals 
appeared  to  be  more  bothered  by  the  necessity  of  monitoring 
aircraft  at  the  edges  of  the  display  which  is  where  the 
possibility  of  incursions  may  be  greatest.  However,  no 
incursions  were  planned  as  part  of  this  study. 
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TABLE  2.  MEAN  POST  RUN  PARTICIPANT  RESPONSES 


Question 


FPL(l) 


Respondent 


Developmental ( 2 ) 


Oculometer 

Oculometer 

Yes 

No 

Yes 

No 

Workload 

5.50 

4.50 

5.00 

3.88 

Performance 

8.33 

7.75 

6.25 

6.88 

Time  Busy 

5.83 

5.25 

5.88 

5.00 

Stress 

4.08 

3.42 

4.88 

3.00 

Interference 

5.25 

1.75 

4.25 

1.50 

Traffic  Volume 

6.67 

5.92 

6.25 

6.13 

Traffic  Composition 

5.83 

2.93 

7.00 

5.92 

Runway  Layout 

2.50 

2.25 

3.88 

2.63 

Aircraft  at  Edges 

3.42 

3.42 

5.13 

4.63 

Metering 

4.42 

4.83 

4.13 

4.00 

Fatigue 

2.42 

1.58 

3.63 

2.13 

These  data  were  initially  analyzed  by  intercorrelating  all  the 
scales  to  include  the  additional  variable  of  the  presence  or 
absence  of  the  oculometer.  Results  indicated  that 
participants  did  find  that  the  presence  of  the  oculometer  was 
annoying.  The  interference  question  correlated  r=. 59 {P< . 00 1 ) 
with  the  presence  or  absence  of  the  oculometer.  Controllers 
responded  in  post-run  interviews  that  they  were  most  aware  of 
the  helmet  mounted  equipment  when  their  task  load  was  low  and 
tended  to  forget  about  it  when  busy.  They  also  indicated  in 
their  questionnaire  that  their  level  of  stress,  r=. 68 (P<. 001 ) , 
was  directly  related  to  their  workload  and  the  percentage  of 
time  they  were  kept  busy,  r=.62 (P<. 001 ) .  Surprisingly, 
controller  responses  on  workload  were  independent  of  their 
self-assessment  of  performance,  r=-.03.  We  often  find  in  this 
type  of  research  at  least  a  mild  inverse  relationship,  but 
these  participants  did  not  see  it  that  way. 

Linear  regression  analyses  were  computed  using  all  the 
questionnaire  variables  against  two  independent  variables;  the 
presence  or  absence  of  the  oculometer  and  the  skill  level  of 
the  participants  (which  was  coded  as  "1"  for  full  performance 
controllers  and  ”2"  for  developmental  or  trainee  controllers). 
The  purpose  of  this  analysis  was  to  determine  the  degree  to 
which  the  questionnaire  itself  could  identify  whether  the 
oculometer  was  in  use  (did  it  impact  participants' 
perceptions)  and  the  impact  of  controller  skill  on 
questionnaire  responses.  Results  on  the  presence  or  absence 
of  the  oculometer  provided  a  multiple  R  of  .697.  An  analysis 
of  variance  (ANOVA)  on  the  regression  was  computed  and  was 
significant  (F=2.41  P<.05).  (The  concept  of  ANOVA  will  be 
explained  in  the  Automated  Performance  Data  section  later  in 
this  report.)  This  meant  that  the  relationship  did  not  occur 
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by  chance.  The  one  variable  that  had  the  greatest  impact  on 
the  regression  was,  not  surprisingly,  the  question  on  the 
level  of  interference  of  the  oculometer. 

A  second  linear  regression  computed  on  skill  as  an  independent 
variable  provided  a  multiple  R  of  .73  with  a  significant  ANOVA 
on  the  regression  (F=2.94,  P<.05).  A  number  of  questionnaire 
items  made  a  significant  contribution  to  this  regression.  The 
reader  will  recall  that  fully  trained  controllers  were  coded 
as  "1"  and  trainees  were  coded  as  ••2".  So  responses  that  have 
positive  correlations  with  the  regression  indicate  that 
trainees  tend  to  rate  them  higher  and  those  with  an  inverse  or 
negative  loading  indicate  that  FPL  controllers  rate  them 
higher.  The  following  is  a  listing  of  significant  beta 
weights  which  reflect  the  contribution  of  the  respective 
variables  to  the  regression  equation  (table  3). 

TABLE  3.  BETA  WEIGHTS  ON  THE  IMPACT  OF  SKILL  ON 
QUESTIONNAIRE  RESPONSES 


Metering 

-.61 

Interference 

-.44 

Fatigue 

.43 

Aircraft  on  Edges 

.54 

This  table  indicates  that  the  controllers  answered  some 
questions  differently  based  on  their  experience.  FPL 
controllers  were  more  concerned  with  the  impact  of  the 
metering  of  in-bound  aircraft  and  the  annoyance  produced  by 
the  oculometer.  The  developmental  controllers  saw  their 
fatigue  and  the  impact  of  having  to  monitor  aircraft  at  the 
edges  of  the  display  as  the  more  important  elements  of  their 
experience . 

As  it  turned  out,  judging  from  the  analysis  of  actual 
performance  to  be  discussed  in  Automated  Performance  Data 
section,  this  was  a  classic  case  of  perceptions  diverging  from 
reality.  However,  despite  what  controllers  thought,  when 
their  actual  performance  was  regressed  against  the  presence  or 
absence  of  the  oculometer,  the  multilinear  regression  provided 
a  multiple  R=.37,  which  was  not  significant.  The  analysis  of 
variance  on  the  regression  was  only  F=.41  (P>.05). 

At  the  end  of  the  questions  that  required  numerical  answers, 
controllers  were  asked  to  describe  their  strategies  for 
working  the  traffic  in  the  simulation.  In  all  the  responses 
provided,  no  one  cited  the  oculometer  as  an  influence  in  their 
operational  strategy  one  way  or  the  other.  The  majority  of 
the  comments  seemed  to  focus  on  the  maintenance  of  horizontal 
separation  using  very  traditional  techniques  of  speed  control 
and  vectors.  The  amount  of  detail  and  complexity  of  the 
operating  strategies  varied  considerably  ranging  from  just  a 
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few  words,  i.e.  "have  the  aircraft  at  compatible  speeds,"  to 
multistep  procedures  such  as:  "ensure  I  had  vertical 
separation  with  the  base  leg  and  downwind  traffic.  Also  make 
sure  the  south  final  aircraft  were  below  my  altitude."  The 
more  complicated  strategies  appear  to  have  been  characteristic 
of  the  more  experienced  controllers. 

Another  question  asked  participants  if  there  was  anything 
which  had  occurred  which  might  have  influenced  the  results. 
There  were  a  number  of  comments  which  related  to  the  operation 
of  the  simulation  itself.  For  example,  there  was  no  automatic 
offset  on  the  data  tags  that  the  participants  had  available  in 
their  home  facility.  Pilot  errors  occurred  and  they  had  to 
take  corrective  action.  While  this  was  not  a  planned  element 
of  the  experiment,  simulation  pilots  do  make  errors,  and,  for 
that  matter,  so  do  real  pilots.  One  controller  complained 
that  the  helmet  hurt  his  head  after  he  wore  it  awhile. 

Another  noticed  a  reduction  in  peripheral  vision  while  wearing 
the  helmet.  Neither  of  these  issues  was  cited  more  than  once. 

AUTOMATED  PERFORMANCE  DATA. 

In  addition  to  the  perceptions  collected  from  the  controller 
participants  by  questionnaire,  this  type  of  research  is  very 
data  rich  in  terms  of  the  availability  of  both  performance 
indicators  and  visual  scanning  data.  Part  of  the  challenge  of 
vision  research  in  an  ATC  simulation  environment  is  deciding 
what  to  analyze  for  the  current  purpose  and  what  to  set  aside 
for  later. 

The  simulator  itself  generates  variables  on  virtually 
everything  that  is  going  on  in  the  "airspace."  Every  time  two 
aircraft  come  too  close  together,  the  information  is  duly 
recorded.  This  can  happen  in  number  of  different  ways.  While 
the  number  of  variables  that  can  be  produced  by  processing  the 
flight  data  of  the  simulated  aircraft  is  limited  primarily  by 
ingenuity  and  computer  time,  there  is  a  good  deal  of 
redundancy  in  these  measures.  A  limited  subset  was  selected 
from  those  available  based  on  experience  with  the  simulation. 
The  following  table  4  is  a  repeat  of  this  list  which  was  also 
presented  earlier  (table  1)  in  the  report  in  the  design 
section. 

In  addition  to  these  so  called  hard  data  variables,  there  was 
input  from  the  over-the-shoulder  observer,  who  rated  the 
performance  (PERFRT)  and  workload  (WLRT)  of  controllers  using 
the  oculometer  on  any  given  run.  It  was  decided  to  integrate 
these  rating  data  into  the  analyses  since  they  provided  input 
from  a  professional  controller  serving  as  observer  and  had 
considerable  face  validity. 
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TABLE  4.  AUTOMATED  PERFORMANCE  MEASURES 


LCNF .  Longitudinal  Conflicts 

LAPI . Longitudinal  API* 

MLAPI . Median  Longitudinal  API 

DLCNF . Duration  Longitudinal  Conflicts 

SCNF . Standard  Conflicts 

SAPI . Standard  Conflict  API 

SAPI . Median  Standard  API 

BCNF . Between  Sector  Conflicts 

BAPI . Between  Sector  API 

MBAPI .  Median  Between  Sector  API 

PREY . Total  Pilot  Key  Strikes 

LAND . Arrival  Landings 


*  API  is  a  measure  of  Conflict  Severity 


The  first  step  in  the  analysis  of  these  data  was  to  inter¬ 
correlate  all  the  performance  variables  to  include  those 
generated  by  the  observer.  Pearson  product-moment 
correlations  were  computed.  These  evaluate  the  degree  to 
which  two  variables  covary  in  relationship  to  the  amount  of 
variance  within  each.  If,  for  example,  variables  A  and  B  are 
perfectly  correlated,  the  result  will  be  a  correlation 
coefficient  of  r=l  or  r=-l  meaning  that  they  each  have  little 
internal  variability  in  comparison  to  how  they  vary  together 
(i.e.,  as  "A"  increases  "B"  increases  at  the  same  pace). 

The  results  of  these  correlations  are  presented  below  in  table 

5. 


TABLE  5.  INTERCORRELATIONS  OF  PERFORMANCE  VARIABLES 


LCNF 

LAPI 

HLAPI 

SCNF 

SAPI 

HSAPI 

Skill 

.29 

.29 

.23 

-.11 

.01 

.28 

TASKLOAD 

.23 

.21 

.14 

.45 

.11 

-.20 

LCNF 

.99 

.75 

.38 

.01 

-.02 

LAPI 

.85 

.39 

.00 

-.03 

HLAPI 

.35 

-.04 

-.06 

SCNF 

.45 

.25 

SAPI 

.59 

HSAPI 

BCNF 

BAPI 

NBAPI 

PKEY 

LAND 


BCNF 

BAPI 

HBAPI 

PKEY 

LAND 

WLRT 

PERFRT 

.67 

.25 

’  .30 

-.08 

.32 

.32 

-.54 

-.27 

-.41 

-.16 

.41 

-.29 

.48 

-.04 

.18 

-.04 

.07 

.45 

,09 

.44 

-.34 

.18 

-.05 

.06 

.43 

.10 

.43 

-.35 

.15 

-.07 

.01 

.29 

.12 

.33 

-.31 

-.22 

-.19 

-.05 

.74 

-.06 

.47 

-.23 

-.01 

-.00 

.02 

.13 

-.39 

.36 

-.18 

.14 

.32 

.34 

.26 

.03 

.38 

-.46 

.33 

.28 

-.44 

.31 

-.01 

-.58 

.85 

-.18 

.32 

-.03 

-.23 

-.01 

.14 

.07 

-.46 

.15 

.50 

-.06 

-.29 

.06 

-.49 


These  correlations  were  based  on  only  those  runs  in  which 
observer  data  were  available  and  would  produce  a  complete 
matrix.  The  observer  only  rated  the  oculometer  position. 
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There  were  no  observer  ratings  on  the  other  position.  Every 
participant  was  rated  on  each  level  of  taskload.  However,  the 
correlations  were  also  computed  for  all  runs  and  participants 
and  did  not  differ  appreciably  for  the  intercorrelations  of 
the  performance  variables  without  the  observer  input. 

Like  most  statistical  techniques,  one  question  that  is  usually 
asked  about  correlation  is  whether  or  not  it  could  have 
occurred  by  chance  given  the  conditions  and  the  participants 
that  were  involved.  With  the  above  table,  the  criterion  for 
whether  a  given  relationship  may  have  existed  beyond  a  chance 
probability  is  determined  by  the  size  of  the  correlation  and 
the  number  of  degrees  of  freedom  involved  in  computing  the 
correlation,  in  this  case  18.  The  correlation  would  have  to 
exceed  r-.44  in  order  to  conclude  that  it  probably  did  not 
occur  by  chance. 

The  intercorrelations  of  performance  variables  provided  two 
kinds  of  information.  First,  it  provides  an  estimate  of 
variables  that  are  redundant.  If  two  variables  correlate 
highly,  then  they  are  basically  telling  us  the  same  thing. 

For  example,  LCNF,  the  nun±)er  of  longitudinal  conflicts, 
correlated  r=.99  with  its  complementary  score  LAPI,  which  is 
the  aircraft  proximity  index  computed  on  the  same  conflicts. 
Besides  giving  the  investigator  a  handle  on  redundant 
variables,  it  provides  first  look  at  relationships  with  key 
variables  such  as  taskload  and  skill. 

Examining  the  top  line  of  the  correlation  matrix,  we  see  the 
computed  values  for  skill  level  against  performance  indicators 
and  observer  ratings.  There  is  a  significant  positive 
relationship  r=.67  between  BCNF,  the  between  sector  conflicts, 
and  skill.  This  is  positive,  because  it  will  be  recalled  that 
the  developmental  controllers  were  arbitrarily  coded  as  2's 
and  the  FPL's  were  I's.  Developmental  controllers  had  more 
between  sector  conflicts  in  the  simulation.  There  was  also  an 
inverse  correlation  r=.54  between  skill  and  the  observers 
performance  rating.  This  meant  that  the  observer  tended  to 
rate  the  performance  of  the  FPL's  higher  than  the 
developmentals .  This  may  have  been  a  self-fulfilling 
prophesy,  however,  because  the  observer  was  not  working  blind 
and  knew  who  was  who  in  the  experiment.  The  second  horizontal 
line  in  the  matrix  was  also  informative.  There  were  two 
significant  correlations.  Taskload  correlated  significantly 
r=.45  against  SCNF,  the  number  of  standard  conflicts,  and 
r=.48  against  the  workload  rating.  It  should  be  noted  that 
these  correlations,  while  they  indicate  a  positive 
relationship  they  do  not  demonstrate  a  strong  relationship 
and,  therefore,  are  only  indicators  that  something  may  be 
going  on. 
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Table  6  sununarizes  the  mean  or  average  scores  for  each  of  the 
performance  variables  and  observer  ratings.  The  reader  will 
note  that  different  scales  apply  and  that  comparisons  within  a 
given  dependent  variable  across  the  experimental  conditions 
described  at  the  left  of  the  table  are  reasonable,  but 
attempting  to  compare  across  variables  is  not. 

Reading  this  complex  table  is  a  challenge  at  the  best  of  times 
and  trying  to  find  meaning  is  even  more  difficult  without  a 
guide.  The  procedure  that  was  used  to  evaluate  whether  or  not 
there  was  meaningful  variance  underlying  the  table  were  two 
very  basic  statistical  techniques  called  analysis  of  variance 
(ANOVA)  and  analysis  of  co-variance  (ANCOVA) ,  respectively. 
What  they  both  do  is  determine  the  probabilities  that  the 
observed  differences  between  the  means  generated  by  the 
different  experimental  conditions  could  have  occurred  by 
chance.  When  differences  are  determined  to  be  significant,  it 
is  interpreted  to  mean  that  it  is  unlikely,  but  not 
impossible,  that  they  occurred  by  chance  alone. 

TABLE  6.  PERFORMANCE  DATA  MEANS 


Experinental 

Condition 


FPL'S 

LCNF 

LAP  I 

HLAPl 

SCNF 

SAP  I 

nSAPl 

BCNF 

BAP  I 

HBAPI 

PKEY 

LAND 

WLRT 

PFRT 

Task  Oeulomater 

Low  No 

0.43 

0.86 

0.71 

1.57 

6.29 

6.14 

3.14 

14.57 

8.14 

1141.86 

9.86 

—  — 

High  No 

1.00 

0.50 

0.50 

2.33 

2.83 

1.00 

3.00 

8.67 

7.50 

1418.83 

9.83 

— 

Low  Yes 

0.00 

0.00 

0.00 

0.83 

1.67 

1.67 

2.50 

11.50 

4.67 

1146.33 

9.67 

5.75 

9.50 

High  Yes 

0.17 

0.17 

0.17 

2.83 

4.67 

1.17 

1.17 

4.33 

4.33 

1420.67 

9.00 

7.00 

9.50 

AU  FPLS 

0.40 

0.40 

0.36 

1.88 

3.96 

2.64 

2.48 

9.96 

6.24 

1276.32 

9.6 

6.38 

9.50 

Developnentals 

Task  Oculometer 

Low  No 

1.00 

1.67 

0.33 

1.33 

2.33 

1.67 

3.33 

14.00 

3.67 

1126.00 

10.33 

— — 

High  No 

1.25 

5.50 

0.50 

5.50 

7.50 

0.75 

1.25 

1.25 

1.25 

1339.00 

9.50 

— 

Low  Yes 

0.25 

0.25 

0.25 

1.25 

4.00 

4.00 

4.25 

18.75 

10.00 

1155.00 

10.50 

6.63 

8.38 

High  Yes 

1.25 

0.75 

0.25 

1.75 

2.50 

2.00 

4.00 

7.75 

6.00 

1322.25 

9.75 

7.75 

8.13 

AU  Devs 

0.93 

2.07 

0.33 

2.53 

4.20 

2.13 

3.20 

10.20 

5.33 

1242.87 

10.00 

7.19 

8.25 

AU  Personnel 

0.60 

1.03 

0.35 

2.13 

4.05 

2.45 

2.75 

10.05 

5.90 

1263.78 

9.75 

6.70 

9.00 

The  first  step  in  the  process  used  to  analyze  the  performance 
data  was  to  compute  a  three-way  ANOVA  on  each  of  the  dependent 
variables.  It  was  initially  a  three-way  analysis  because  of 
three  independent  variables;  skill  level,  taskload,  and 
oculometer  presence  or  absence.  Table  7  lists  the  results  of 
these  analyses  by  dependent  variable. 
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TABLE  7. 


THREE-WAY  ANOVA' S  OF  PERFORMANCE  VARIABLES 


Variable 

Main  Effects 

Interactions 

LCNF 

None 

None 

LAP  I 

None 

None 

MLAPI 

None 

None 

SCNF 

Taskload* 

None 

SAP  I 

None 

None 

MS  API 

None 

None 

BCNF 

None 

Skill  by  Oculometer 

BAP  I 

Taskload* 

None 

MB  API 

None 

Skill  by  Oculometer 

PKEY 

Taskload* 

None 

LAND 

None 

None 

WLRT 

Taskload* 

None 

PFRT 

Skill* 

None 

*  P<.05 

This  table  shows  several  facts.  A  main  effect  is  the  result 
of  one  of  the  independent  variables.  While  it  is  possible  to 
obtain  a  significant  main  effect  in  a  design  that  also  has  an 
interaction,  it  is  not  possible  to  determine  what  it  means 
without  breaking  down  the  interaction.  On  the  performance 
variables  in  this  study  there  were  no  significant  main  effects 
in  variables  that  had  an  interaction.  An  interaction 
indicates  that  the  results  of  the  independent  variables,  as 
seen  in  the  measurement  of  the  dependent  variable,  were  not 
directly  additive.  For  example,  on  the  surface  it  appears 
that  for  BCNF,  there  were  no  main  effects  from  taskload, 
skill,  or  oculometer  presence.  However,  the  interaction 
indicates  that  in  order  to  interpret  the  impact  of  skill,  for 
example,  you  must  evaluate  results  with  and  without  the 
oculometer  separately. 

In  order  to  understand  the  results  in  table  7,  the  reader  is 
encouraged  to  also  examine  table  6,  Performance  Data  Means. 

The  following  is  the  interpretation  of  each  of  the  variables, 
in  turn,  which  had  either  main  effects  or  interactions.  SCNF 
is  the  first  variable  to  show  significance.  Results  indicate 
that  there  were  more  standard  conflicts  at  the  higher  taskload 
regardless  of  whether  the  oculometer  was  in  use  or  what  the 
skill  of  the  controller  was  working  the  airspace.  For  BCNF, 
it  was  more  complicated  because  there  was  a  skill  by 
oculometer  interaction  {F=10.09,  P<.01).  This  was  handled 
first  by  simplifying  the  situation,  then  by  running  an 
analysis  of  covariance  and  removing  any  contribution 
contributed  by  taskload.  In  table  8  are  the  adjusted  means 
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for  the  contributions  of  skill  and  oculometer  on  BCNF  between 
sector  conflicts. 

TABLE  8.  ADJUSTED  MEANS  FOR  BCNF 

Oculometer 

Skill  Yes  No 

FPL  1.83  3.04 

Developmental  4.12  2.20 

In  order  to  evaluate  where  significant  differences  actually 
exist  between  the  cells  of  this  design,  post-hoc  comparisons 
had  to  be  made.  Since  each  of  these  comparisons  involved  only 
two  levels,  it  was  simplest,  given  the  computer  tools 
available,  to  go  directly  to  a  technique  called  the  Newman- 
Keuls  analysis,  which  makes  these  comparisons  allowing  for  the 
number  of  levels  involved.  The  following  represents  the 
results  for  the  specific  comparisons  on  BCNF  (table  9). 

TABLE  9.  SPECIFIC  COMPARISONS  FOR  BCNF  DATA 

Variable  Significant? 

Skill  -  No  Oculometer  NS 

Skill  -  Yes  Oculometer  * 

Oculometer  -  FPL's  * 

Oculometer  -  Developmentals  * 

*  P  <  .05,  NS=  Not  Significant 

The  only  combination  that  was  not  significant  was  the  impact 
of  skill  when  there  was  no  oculometer  in  use.  When  using  the 
oculometer,  developmental  controllers  made  significantly  more 
errors  than  did  the  FPL's.  Ironically,  the  FPL's  made  more 
errors  without  the  oculometer  than  with  it.  One  might 
speculate  that  the  annoyance  they  reported  from  the  equipment 
encouraged  them  to  pay  more  attention  to  the  airspace  between 
the  sectors. 

The  next  performance  variable  which  showed  significance  was 
BAPI.  This  was  the  aircraft  proximity  index  computed  for  the 
between  sector  conflicts.  Only  taskload  was  significant  (F= 
7.46,  P<  .01).  Referring  back  to  the  table  of  means,  the 
results  of  BAPI  are  very  interesting  when  evaluated  in  terms 
of  taskload.  Regardless  of  skill  or  oculometer  presence,  when 
taskload  increased,  the  severity  of  between  sector  conflicts, 
as  measured  by  BAPI,  decreased.  Again,  higher  taskload  may 
have  lead  to  greater  vigilance. 

The  next  variable  to  reach  significance  was  MBAPI.  This  was 
the  median  between  sector  API  scores  instead  of  using  means  as 
in  BAPI.  The  three  way  ANOVA  produced  a  skill  by  oculometer 
interaction  (F“5.43,  P<.05).  As  with  the  previous  situation 
with  the  BCNF  variable,  it  was  decided  to  simplify  the 
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situation  by  computing  an  ANCOVA  and  removing  any  variance 
that  might  have  been  contributed  by  the  taskload.  The  ANCOVA 
also  produced  a  significant  skill  by  oculometer  interaction 
(F=5.89,  P<.05).  The  adjusted  means  for  the  remaining  design 
are  listed  below  in  table  10. 

TABLE  10.  ADJUSTED  MEANS  FOR  MBAPI 

Oculometer 


Yes 

No 

Skill 

FPL 

4.50 

7.78 

Developmental 

8.00 

2.40 

While  it  would  appear  that  there  are  considerable  differences 
here  based  on  skill  and  the  presence  of  the  oculometer  or  its 
absence,  post-hoc  testing  using  the  Newman-Keuls  procedure 
does  not  bear  this  out.  It  only  takes  one  difference  to  drive 
the  interaction.  Table  11  presents  the  results  of  the  post- 
hoc  analysis. 


TABLE  11.  POST-HOC  ANALYSIS  OF  MBAPI 


Variable 

Skill  -  No  Oculometer 
Skill  -  Yes  Oculometer 
Oculometer  -  FPL's 
Oculometer  -  Developmental s 
*P<  .05  NS=Not  Significant 


Significant? 

* 

NS 

NS 

NS 


Without  the  oculometer,  FPL's  had  more  severe  between  sector 
conflicts  than  did  the  developmental  controllers.  Were  they 
overconfident?  One  might  speculate  that  the  developmentals 
attempted  to  create  larger  buffers  between  aircraft  due  to 
their  lack  of  experience  in  the  airspace;  so  when  they  had  a 
conflict,  it  was  less  severe  as  measured  by  the  median  between 
sector  API  scores. 


Why  were  the  other  possible  paired  comparisons  in  table  11  not 
significant?  There  are  a  number  of  statistical  issues. 

First,  the  Newman-Keuls  technique  is  fairly  conservative  and 
attempts  to  control  the  probability  that  significance  will  be 
found  where  none  exists.  It  computes  a  critical  difference 
necessary  for  each  comparison  that  it  makes  taking  into 
consideration  the  internal  variability  of  each  cell  in  the 
design  and  the  degrees  of  freedom,  which  are  determined  in 
part  by  sample  size.  Since  the  number  of  developmental 
controllers  was  less  than  the  number  of  FPL's,  there  was  less 
statistical  power  available  for  the  comparison  of 
developmentals  across  the  oculometer  variable.  Judging  from 
the  table  of  means  alone,  it  appears  that  it  should  have  been 
significant.  By  using  a  less  conservative  test,  such  as  the 
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"t”  test,  significance  could  have  been  achieved  for  the 
developmentals  across  the  levels  of  oculometer.  This  would 
have  been  an  attempt  to  pursue  something  that  may  well  not 
exist,  and  it  was  not  reported  here. 

The  last  three-way  ANOVA  to  demonstrate  any  significance  was 
the  analysis  on  PKEY,  the  frequency  of  pilot  keystrokes  in 
response  to  controller  inputs  via  the  simulated  radio  channel. 
PKEY  is  an  effective  indicator  of  how  active  the  controllers 
are  on  the  radio.  In  the  current  design,  only  one  main  effect 
was  significant  and  this  was  taskload  (F=7.67,  P<.01). 
Referring  back  to  table  6,  the  reader  can  see  quite  clearly 
that  it  did  not  matter  who  the  controllers  were  or  whether  or 
not  they  were  using  the  oculometer.  Higher  taskload,  which 
was  part  of  the  research  design,  led  to  higher  frequencies  of 
keystrokes  by  the  pilots  and,  undoubtedly,  a  higher  frequency 
of  radio  instructions  from  the  controller  to  the  simulation 
pilots. 

The  last  performance  data  analyses  were  conducted  on  the 
observer's  ratings.  Since  he  only  rated  the  oculometer 
position,  the  design  for  this  analysis  involved  only  two 
factors:  taskload  and  skill.  Two-way  ANOVA 's  were  computed 
for  the  performance  (PERT)  and  the  workload  (WLRT)  ratings, 
respect ivey.  As  indicated  earlier  in  table  11,  both  of  these 
analyses  resulted  in  one  significant  main  effect  each.  For 
WLRT,  the  significant  main  effect  was  taskload  (F=5.33, 

P<.05).  Higher  task  loads  were  viewed  by  the  observer  as 
producing  higher  workloads.  Since  this  was  not  a  blind 
rating,  however,  the  observer  knew  which  runs  were  operated  at 
each  of  the  two  levels  of  taskload,  so  his  ratings  may  have 
been  somewhat  confounded.  For  PFRT,  no  other  independent 
variables  influenced  the  results  other  than  the  skill  level  of 
the  participant.  FPL's  were  invariably  rated  higher  than  were 
the  developmentals  (F=6.71,  P<.05).  This  too  could  have  been 
confounded  because  the  observer  knew  who  was  who.  The 
observer  was  a  professional  and  doubtlessly  tried  to  maintain 
his  objectivity  to  the  extent  humanly  possible. 

EYE  MOVEMENT  DATA. 


There  were  two  basic  vision  variables  with  a  number  of  ways  of 
looking  at  each.  These  were  saccades  or  movements  between 
fixations  and  the  fixations  themselves.  Fixations  have 
frequency  and  duration,  and  saccades  have  frequency  and 
magnitude.  An  additional  variable  called  eye  motion  workload 
was  created  by  an  engineer,  George  Hetrich,  at  the  FAA 
Technical  center.  It  was  computed  by  taking  the  average 
saccade  motion  in  degrees  and  dividing  by  the  number  of 
saccades.  The  larger  the  number,  the  more  the  eyes  moved  on 
the  average  between  fixations.  The  oculometer  generated  basic 
data  concerning  the  saccades  and  fixations.  Then,  through 
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statistical  processing,  additional  variables  were  created 
against  the  possibility  that  each  would  add  something  to  the 
knowledge  of  the  scanning  process.  What  follows  are  the 
definitions  of  the  variables  that  will  be  seen  abbreviated  in 
subsequent  analyses  (table  12). 


TABLE  12.  DEFINITIONS  OF  VISION  VARIABLES 


FIXMEAN 

FIXMDN 

SACDURM 

SACDURMD 

SACMAGMN 

SACMDN 

PUPIL  MEAN 

PUPIL  MDN 

FIXFREQ 

VISEFFIC 

EYEMOTWL 

PUPILMOTWL 


Average  Fixation  Duration  (Seconds) 

Median  Fixation  Duration  (Seconds) 

Average  Saccade  Duration  (Seconds) 

Median  Saccade  Duration  (Seconds) 

Average  Saccade  Magnitude  (Degrees  of  Angle) 
Median  Saccade  Magnitude  (Degrees  of  Angle) 
Average  Pupil  Size  (ASL  Units) 

Median  Pupil  Size  (ASL  Units) 

Average  Fixation  Frequency  (Count) 

Visual  Efficiency 
Eye  Motion  Workload 
Pupil  Motion  Workload 


The  last  three  variables  in  table  12  were  developed  by  George 
Hetrich  who  was  doing  some  of  the  data  reduction  analyses  at 
the  FAA  Technical  Center.  Visual  efficiency  is  the  proportion 
of  the  total  scanning  time  that  is  spent  fixating.  It  is 
computed  using  the  following  formula;  Average  Fixation 
Duration  *  Fixation  Frequency/ (Average  Fixation  Time  * 

Fixation  Frequency)  +  (Average  Saccade  Duration  *  Saccade 
Frequency).  This  provides  a  fraction  from  0  to  1.  Eye  motion 
workload  is  the  average  degrees  per  second  that  the  eyes  moved 
during  the  entire  scanning  period.  This  is  computed  by  the 
formula:  Average  Saccade  Magnitude  *  Saccade  Frequency /Total 
Time  in  seconds.  This  provides  a  number  in  degrees  per  second 
for  each  1/2  hour  simulation.  The  last  Hetrich  measure  was 
pupil  motion  workload.  This  refers  to  the  change  in  pupil 
diameter  over  time.  Pupil  diameter  was  measured  at  each 
fixation.  Pupil  motion  workload  was  computed  by  cumulating 
the  absolute  value  of  the  differences  between  each  pair  of 
fixation  records.  These  pupil  diameter  values  were  then 
averaged  for  the  total  run  by  dividing  by  the  total  time  in 
the  simulation.  The  result  was  in  ASL  units,  an  arbitrary 
metric  developed  by  the  manufacturer  of  the  oculometer.  This 
was  then  converted  to  units  of  millimeters  per  second  by 
applying  a  conversion  formula  using  .044  millimeters  per  ASL 
unit,  which  was  the  ratio  provided  by  the  manufacturer. 

A  summary  of  all  the  vision  related  data  to  include  variables 
which  are  derivations  of  the  more  basic  saccades  and 
fixations  is  presented  in  table  13. 
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TABLE  13.  VISUAL  DATA  MEANS 


Experi Mental 

Condition  Vision  Variables 


FIX 

FIX 

SAC 

SAC 

SAC 

SAC 

PUPIL 

PUPIL 

FIX 

VIS 

EYE 

PUPIL 

MEAN 

HDN 

DME 

PHD 

HGHE 

HSHD 

HEAN 

HDN 

ATF 

EFFI 

HOT ION 

HOT ION 

FPL's 

Task 

Low 

0.42 

0.31 

0.11 

0.03 

5.23 

2.70 

126.83 

127.Q3 

3780.33 

0.80 

10.28 

3.05 

High 

0.47 

0.36 

0.09 

0.03 

5.92 

3.23 

129.33 

131.00 

3567.00 

0.84 

10.87 

2.30 

All  FPL's 

0.41 

0.33 

0.10 

0.03 

5.57 

2.96 

128.08 

129.42 

3673.66 

0.82 

10.57 

2.67 

DevelopMentals 

Task 

Low 

0.44 

0.33 

0.12 

0.03 

5.85 

3.18 

125.00 

126.75 

3456.75 

0.78 

10.61 

5.58 

High 

0.53 

0.39 

0.12 

0.02 

5.34 

3.00 

119.25 

122.00 

3085.50 

0.80 

9.11 

1.91 

All  Devs 

0.48 

0.36 

0.12 

0.02 

5.59 

3.09 

122.13 

124.38 

3271.13 

0.79 

9.86 

3.75 

All 

0.44 

0.34 

0.11 

0.03 

5.58 

3.02 

125.70 

127.40 

3512.65 

0.81 

10.29 

3.10 

The  diversity  of  units  of  measurement  is  obvious  from  this 
table.  Also,  a  cursory  glance  within  the  different  variables 
and  across  task  loads  and  skill  levels  are  not  very 
informative.  There  appears  that  there  might  be  a  difference 
in  fixation  frequencies  between  the  FPL's  and  the 
developmentals,  but  not  much  more  information  is  available 
just  from  looking  at  the  table  itself. 

The  next  step  in  the  analysis  was  to  search  for  redundancy 
among  the  vision  variables.  Since  some  were  the  product  of 
the  same  sources  and  were  simply  different  ways  of  looking  at 
the  same  information,  it  seemed  likely  that  they  would  be 
closely  related.  This  was  accomplished  by  intercorrelating 
all  the  vision  variables.  Table  14  provides  the  Pearson 
product -moment  correlations  of  the  vision  variables  along  with 
their  relationships  to  the  observer's  ratings  of  WLRT  and 
PFRT.  As  in  the  previous  tables,  any  correlation  above  the 
absolute  value  of  .44  is  significant  from  zero.  This  does  not 
necessarily  mean  that  a  correlation  of  .44  accounts  for  a 
great  deal  of  common  variance,  only  that  it  probably  did  not 
occur  by  chance.  In  terms  of  redundancy,  there  were  a  number 
of  strong  relationships  such  as  the  one  between  FIXMEAN  and 
FIXMDN  and  between  PUPILMEAN  and  PUPILMDN.  Relationships  like 
these  with  correlations  of  r=.99  and  r=1.0,  respectively, 
indicate  that  the  variables  involved  are,  for  most  purposes, 
virtually  interchangeable. 
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TABLE  14.  INTERCORRELATIONS  OF  VISION  VARIABLES 


FIX 

FIX 

SAC 

SACO 

SAC 

SAC 

PUPIL 

PUPIL 

FIX 

VISE 

EVE 

PUPIL 

PFRT 

MEAN 

HON 

DURM 

URHO 

HAGHN 

HON 

MEAN 

nON 

FREQ 

FFIC 

nOTWL 

HOTUL 

ULRT  -.49 

.14 

.14 

-.47 

-.59 

.11 

.16 

-.16 

-.15 

.10 

.41 

.15 

-.15 

PFRT 

.41 

.42 

-.16 

.18 

-.12 

.04 

.18 

.18 

-.37 

.31 

-.47 

-.21 

FIXMEAN 

.99 

-.20 

-.41 

.28 

.59 

.59 

.63 

-.87 

.58 

-.62 

-.01 

FIXHDN 

-.23 

-.43 

.25 

.57 

.55 

.58 

-.86 

.61 

-.63 

-.04 

SACDURM 

.75 

.05 

-.09 

.06 

.06 

-.25 

-.90 

-.11 

.08 

SCADURHD 

-.23 

-.38 

-.20 

-.21 

.01 

-.79 

-.12 

.03 

SACMAGHN 

.91 

.73 

.71 

-.37 

.06 

.46 

.02 

SACMON 

.81 

.81 

-.59 

.32 

.14 

.07 

PUPILFIEAN 

1.00 

-.62 

.20 

-.01 

.19 

PUPILNDN 

-.65 

.22 

-.05 

.19 

FIXFREQ 

-.17 

.64 

-.00 

VISEFFIC 

-.19 

-.02 

EYEHOTUL 

-.04 

Due  to  the  results  of  the  intercorrelations  of  the  vision  data, 
it  was  decided  to  carry  out  an  additional  analysis  using  a 
technique  called  Factor  Analysis.  This  procedure  examines  the 
matrix  of  correlations  and  systematically  looks  for  clusters  of 
variables  that  appear  to  be  so  closely  related  that  they  actually 
form  new  variables  called  factors.  The  process  of  Factor 
Analysis  is  more  systematic  than  visually  examining  a  correlation 
matrix  and  final  results  appear  in  table  15.  These  are  Factor 
Loadings  which  are  correlations  of  the  variables  with  the 
factors. 

The  table  indicates  that  of  all  the  vision  variables  there  are 
basically  four  factors  operating  in  the  data  base  that  is 
currently  present. 

Factors  are  usually  named  based  on  the  variables  which  make  them 
up.  The  first  factor  contained  variables  that  related  to  the 
size  of  the  average  saccades  and  the  average  size  of  the 
participant's  pupil.  Why  these  are  related  is  unclear  at  this 
time.  The  second  factor  contained  variables  that  were  highly 
correlated  and  represented  different  measures  of  saccade 
duration.  In  addition,  the  visual  efficiency  variable  loaded  on 
factor  2.  This  is  not  difficult  to  understand,  since  visual 
efficiency  is  computed  using  saccade  data  as  a  component  of  the 
numerator  of  the  formula.  The  third  factor  concerns  primarily 
fixation  data  to  include  the  fixation  means  and  medians  along 
with  the  fixation  frequency.  Eye  motion  workload  also  loaded  on 
that  factor.  It  has  saccade  frequency  as  a  major  component  of 
its  formula;  this  is  directly  related  to  fixation  frequency  since 
each  saccade  ends  in  a  fixation.  The  fourth  factor  was  pupil 
motion  workload  by  itself.  As  shown  in  the  correlation  matrix  it 
did  not  correlate  with  any  other  vision  variable  and,  therefore, 
formed  its  own  factor. 
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TABLE  15.  FACTOR  LOADINGS  FOR  VISION  VARIABLES 


Factor 


Magnitude /Pupil  Duration  Fixations  PUPMOTWL 


Variable 

FIXMEAN 

-.876 

FIXMDN 

-.880 

SACDURM 

.992 

SACDURMD 

.880 

SACMAGMN 

.989 

SACMDN 

.944 

PUPMEAN 

.795 

PUPMDN 

.780 

FIXFREQ 

.885 

VISEFFIC 

-.945 

EYEMOTWL 

.894 

PUPMOTWL 

Pearson  product -moment  correlations  were  also  computed  between 
the  performance  variables  to  include  over  the  shoulder  ratings  by 
a  subject  matter  expert  seen  above  and  the  vision  data  collected 
with  the  oculometer.  Table  16  provides  all  of  these 
correlations.  This  table  contains  a  great  deal  of  data  and  is 
rather  visually  noisy. 

The  easiest  way  to  read  this  table  is  to  scan  across  each  row  of 
the  performance  variables  and  attempt  to  identify  those 
relationships  which  exceed  the  critical  value  of  r=.44.  This  is 
the  minimum  correlation  that  would  be  significant  from  zero. 

TABLE  16.  CORRELATIONS  BETWEEN  PERFORMANCE  AND  VISION 


PERF 

FIX 

FIX 

SAC 

SACO 

SAC 

SAC 

PUPIL 

PUPIL 

FIX 

VISE 

EYE 

PUPIL 

RTG 

NEAN 

NON 

BURN 

URNO 

NAGNN 

NON 

BEAN 

NON 

FREQ 

FFIC 

HOTUL 

HOTUL 

Skill 

-.54 

.11 

.14 

.24 

-.20 

.01 

.05 

-.15 

-.12 

-.22 

-.13 

-.15 

.09 

TASKLOAO 

-.04 

.24 

.29 

-.12 

-.20 

.08 

.11 

-.02 

.00 

-.16 

.20 

-.05 

-.16 

LCNF 

-.34 

-.05 

-.01 

-.24 

-.17 

-.06 

-.05 

-.10 

-.11 

.10 

.19 

.04 

.10 

LAP  I 

-.35 

-.03 

.01 

-.22 

-.17 

-.01 

.00 

-.02 

-.04 

.06 

.17 

.06 

.07 

HLAPI 

-.31 

.04 

.05 

-.11 

-.13 

.14 

.15 

.21 

.19 

-.04 

.08 

.09 

-.05 

SCNF 

-.23 

.09 

.09 

-.32 

-.29 

-.02 

.06 

.03 

.02 

.06 

.29 

-.03 

.15 

SAPI 

-.18 

-.14 

-.13 

-.15 

-.05 

-.29 

-.28 

-.29 

-.29 

.16 

.08 

-.12 

-.11 

MAPI 

-.46 

-.21 

-.22 

-.19 

-.25 

-.13 

-.17 

-.19 

-.19 

.30 

.07 

.16 

-.05 

BCNF 

-.58 

-.14 

-.15 

.47 

.02 

.22 

.12 

.09 

.09 

-.12 

-.42 

.12 

.34 

BAP  I 

-.23 

-.03 

-.01 

-.16 

-.34 

.24 

.23 

.17 

.16 

.06 

.13 

.24 

.05 

HBAPI 

-.46 

-.23 

-.19 

-.19 

-.34 

.14 

.07 

-.02 

-.02 

.30 

.08 

.40 

.19 

PKEY 

-.06 

.06 

.09 

-.57 

-.43 

-.28 

-.17 

-.17 

-.18 

.26 

.47 

-.03 

.03 

LAND 

.06 

.18 

.16 

.22 

.05 

-.11 

-.02 

.20 

.20 

-.26 

-.11 

-.30 

.31 

ULRT 

-.49 

.14 

.14 

-.47 

-.59 

.11 

.16 

-.16 

-.15 

.10 

.41 

.15 

-.15 

PFRT 

1.00 

.41 

.42 

-.16 

.18 

-.12 

.04 

.18 

.18 

-.37 

.31 

-.47 

-.21 
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Beginning  with  skill,  only  performance  rating  was  significant. 

It  was  an  inverse  relationship  because  of  the  way  the 
participants  were  coded:  FPL  1  and  Developmental  2.  No  vision 
variables  correlated  significantly  with  either  skill  or  taskload. 
This  held  true  for  the  next  six  performance  variables.  BCNF 
correlated  against  saccade  duration  means  (SACDURMN).  BAPI  and 
MBAPI  did  not  correlate  against  any  performance  variables.  PKEY 
was  the  measure  of  the  pilots  key  strokes  in  response  to 
controller  verbal  instructions.  It  correlated  with  three  vision 
variables:  SACDURM,  SACDURMD,  and  VISEFFIC.  These  three 
variables  all  contain  information  related  to  saccade  duration. 

It  appears  that  controller  activity  level  is  inversely  related  to 
saccade  durations;  the  busier  he  becomes,  the  shorter  and  more 
frequent  are  his  saccades.  Workload  rating  by  the  over  the 
shoulder  observer  was  also  inversely  related  to  saccade 
durations.  The  shorter  the  saccades  the  higher  the  perceived 
workload  as  rated  by  the  observer.  It  is  unlikely  that  the 
observer  was  keying  in  on  the  eye  movements  themselves,  but 
rather  he  was  focusing  on  the  situational  demands  of  the 
environment  created  by  the  simulation.  Eye  motion  workload 
EYEMOTWL  was  inversely  related  to  performance  ratings.  The 
reader  will  recall  that  EYEMOTWL  loaded  on  the  same  factor  as  the 
fixation  frequency  and  duration  measures  although  it  contains 
saccade  information.  It  may  well  be  more  sensitive  to 
performance  rating  than  the  fixation  data,  which  did  not 
correlate  significantly  with  the  PFRT. 

One  of  the  principle  issues  in  this  study  was  whether  or  not 
fully  trained  controllers  behaved  differently  than  developmental 
controllers.  Unfortunately,  because  of  the  sampling  used,  these 
two  groups  of  people  differed  more  in  terms  of  their  familiarity 
with  the  specific  airspace  they  were  working  than  based  on  years 
of  experience  (i.e.,  most  of  the  developmentals  had  come  from 
other  facilities).  Regression  analysis  was  employed  to  determine 
whether  there  were  vision  variables  that  could  separate  the 
participants  based  on  their  skill  levels:  FPL's  1  and 
Developmentals  2.  Regression  offers  more  power  in  looking  for 
relationships  than  do  simple  bivariate  correlations  just 
discussed.  It  provides  an  opportunity  to  maximize  the 
contributions  of  number  of  predictor  variables  (in  this  case 
vision  variables)  against  a  criterion  variable.  The  criterion 
here  was  skill  level,  which  unfortunately  was  a  dichotomous 
variable.  It  is  more  difficult  to  obtain  a  viable  relationship 
against  a  dichotomy  than  against  a  continuous  variable  which  has 
more  information.  It  would  have  been  better  if  skill  could  have 
been  graded  along  some  sort  of  continuum.  However,  the 
regressions  were  computed  with  what  we  had. 

Applying  regression  analysis  to  data  like  this  might  be  likened 
to  applied  chemistry,  you  try  alternative  mixtures  until 
something  interesting  occurs.  In  regression,  you  have  a  number 
of  alternatives.  The  first  is  standard  regression.  This  enters 
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all  the  predictor  variables  and  produces  an  equation  which  shows 
how  they  fit  against  the  criterion.  Standard  regressions  were 
run  against  skill,  and  nothing  was  significant.  Standard 
regressions  throw  all  the  variance  available  into  the  pot,  and 
with  it  comes  considerable  error  variance  along  with  the 
relationship  information  of  interest.  Another  alternative  is  a 
step  wise  regression  in  which  the  computer  evaluates  the 
potential  contribution  of  each  predictor  and  steps  them  into  the 
equation  until  it  is  told  to  stop,  usually  at  a  point  where  there 
is  certain  amount  of  error  remaining  in  the  system.  The 
researcher  can  also  tell  the  computer  to  stop  after  a  certain 
number  of  steps.  The  result  will  be  a  regression  limited  to  a 
given  number  of  variables.  This  was  the  process  that  was 
employed.  Based  on  what  amounted  to  trial  and  error,  it  was 
determined  that  the  strongest  regressions  could  be  achieved  using 
step  wise  regression  limited  to  six  steps. 

Regressions  were  computed  against  skill,  taskload,  workload 
rating  and  performance  ratings.  Table  17  describes  the  overall 
results  of  those  analyses. 


TABLE  17.  REGRESSION  ANALYSES  ON  VISION  VARIABLES 
Criterion  Variable  Multiple  R  ANOVA  on  Regression 


Skill 

.857 

F=5.59 

♦  * 

Taskload 

.464 

F=2.33 

NS 

Workload  Rating 

.710 

F=5.41 

Performance  Rating 

.563 

F=2.49 

NS 

*  P<.05 
**  P<.01 

NS-Not  Significant 


Selected  vision  measures  produced  a  significant  regression  for 
skill  as  a  dependent  variable.  The  regressions  was  also 
significant  for  workload  ratings  but  not  for  taskload  or 
performance  ratings.  The  lack  of  any  workable  regression  for 
taskload  was  not  too  surprising  since  it  did  not  correlate  well 
with  any  of  the  vision  variables,  indicating  that  by  itself 
taskload  did  not  have  any  noticeable  impact  on  the  vision  results 
and  performance  ratings,  as  a  whole,  were  not  predictable  based 
on  vision  data  alone. 

Table  18  provides  the  beta  weights  for  each  of  the  significant 
regressions.  The  beta  weight  is  an  indicator  of  the  degree  to 
which  each  variable  that  entered  the  regression  was  successful  in 
accounting  for  variance  in  the  criterion  variables. 
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TABLE  18.  BETA  WEIGHTS  ON  SIGNIFICANT  VISION  VARIABLE 
REGRESSIONS 


Criterion 

Vision  Variable 

Beta  Weicht 

Skill 

FIXFREQ 

-1.77 

SACDURMD 

-1.29 

PUPMEAN 

-  .71 

Workload  Rating 

PUPMEAN 

-  .68 

SACDURMD 

-  .53 

The  sign  of  the  beta  weight  is  informative  in  that  it  indicates 
whether  the  variable  should  be  added  to  or  subtracted  from  the 
equation.  Variables  with  a  negative  sign  lower  the  predicted 
value  of  the  criterion  variable  towards  skill  1  and  predict  FPL, 
while  those  with  a  positive  sign  increase  the  value  of  the 
equation  and  predict  towards  a  2  or  developmental  controller. 

For  example,  fixation  frequency  under  skill  has  a  negative  beta 
weight.  Referring  back  to  table  13  under  fixation  frequency,  it 
appears  that  FPL's  have  a  higher  mean  of  fixations.  The 
regression  indicates  that  higher  fixations  should  predict  lower 
controller  skill  code,  in  other  words,  an  FPL  controller. 

When  examining  the  means  of  the  variables  that  entered  the 
regression,  it  does  not  appear  as  though  there  can  be  much 
difference  between  the  groups  because  of  the  nature  of  the 
measurement.  With  the  exception  of  fixation  frequency,  in  which 
it  is  clear  that  developmental  controllers  have  fewer  fixations, 
the  other  measures  are  more  subtle.  Saccade  duration  is  in 
milliseconds.  Saccade  magnitude  is  in  degrees  of  rotation,  and 
it  does  not  take  much  of  a  rotation  to  change  things  in  the 
person's  field  of  few.  Things  happen  fast  with  eye  movements. 

The  results  of  this  analysis  indicate  that  full  performance  level 
controllers  search  their  environment  more  frequently  than  their 
less  experienced  colleagues. 

Judging  from  the  correlation  of  workload  ratings  and  saccade 
duration  medians  SACDURMD  of  -.59  and  the  regression  in  which 
pupil  mean  and  SACDURMD  both  loaded  negatively,  it  appears  that 
lower  workload  ratings  are  associated  with  longer  saccade 
durations,  shorter  magnitudes,  and  larger  pupil  sizes.  The 
observer  apparently  was  keying  on  dimensions  of  workload  that 
could  be  associated,  in  part,  with  the  controllers  visual 
activity.  It  is  unlikely  that  the  observer  was  using  this 
information  directly  and  was  probably  cuing  on  other  variables. 

To  this  point  in  the  analysis,  the  data  has  been  dealt  with  in  a 
specific  manner.  Due  to  the  large  volume  of  data  generated  by 
the  oculometer,  it  was  necessary  to  break  the  information  into 
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reasonable  chunks  for  further  evaluation.  These  blocks  of  data 
were  in  30-minute  parcels,  which  were  summarized  during  the 
initial  data  reduction  by  computing  descriptive  statistics,  such 
as  means  and  medians,  and  then  using  these  descriptives  to 
represent  the  parcel  or  block  from  which  they  arose.  The 
descriptive  then  became  the  number  which  entered  subsequent 
analyses.  This  has  the  advantage  of  greatly  simplifying  the 
process  and  the  disadvantage  of  discarding  a  great  deal  of 
variance  that  could  be  useful  if  there  were  analytic  resources  to 
deal  with  it. 

In  order  to  evaluate  the  possibilities  using  a  smaller  data  chunk 
size  given  the  resources  available,  a  select  number  of  variables 
were  chosen  and  were  reduced  to  the  5-minute  level.  The  data 
that  were  then  statistically  analyzed  were  the  descriptive 
statistics  for  each  of  the  5-minute  blocks  within  each  of  the  30- 
minute  simulations.  The  variables  chosen  were  saccade  magnitude 
(SACMAGMN),  saccade  duration  (SACDURM),  fixation  frequency 
(FIXFREQ),  and  fixation  duration  (FIXMEAN). 

The  analyses  of  these  four  variables  examined  the  potential 
impact  of  skill,  taskload,  and  time  sequence  on  the  vision  data. 
The  data  for  the  variables  were  plotted  and  are  presented  in 
figures  3  through  6.  For  both  saccade  magnitude  and  duration, 
the  plots  took  on  a  characteristic  shape  indicating  what  appeared 
to  be  a  decline  over  time.  A  three-way  ANOVA  was  computed  on 
saccade  magnitude  with  taskload,  skill,  and  time  as  independent 
variables.  There  were  no  significant  interactions  or  main 
effects  for  taskload  or  skill  (P>.05).  However,  there  was  a 
significant  effect  of  time  (F=165.11,  P<.01).  Deleting  the  first 
time  block  and  running  the  analysis  again  led  to  a  significant 
time  main  effect,  but  of  smaller  magnitude  (F=16.93,  P<.01). 

With  saccade  duration  the  results  were  similar.  There  were  no 
interactions  and  only  one  main  effect,  time  when  including  all 
six  time  blocks  (F=10.52,  P<.01).  When  the  first  time  block  was 
removed,  then  the  time  main  effect  was  no  longer  significant 
(F=.568,  P>.05).  It  is  apparent  that  over  the  first  5  to  10 
minutes  of  scanning  there  are  significant  changes  in  saccade 
magnitude  and  duration.  Both  decreased  then  stabilized  for  the 
remainder  of  the  30-minute  shift.  It  is  well  known  that 
controller  operational  errors  most  frequently  occur  at  the 
beginning  of  a  shift  or  right  after  a  break. 

Fixation  frequencies,  as  described  in  figure  5,  show  no 
characteristic  pattern.  No  interactions  or  main  effects  were 
significant  including  time  (P>.05).  The  graph  seems  to  indicate 
that  developmental  controllers,  when  under  high  task  load,  have 
lower  fixation  frequencies.  They  appear  to  be  scanning  less,  but 
the  ANOVA  did  not  demonstrate  this  statistically.  This  could  be 
due,  in  part,  to  the  complex  variability  in  the  design  coupled 
with  the  multiple  sources  of  measurement  error  when  analyzing  the 
data  as  a  whole.  A  little  known  nonparametric  test  was  applied 
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FIGURE  5.  MEAN  FIXATION  FREQUENCIES 


to  the  data  within  each  time  block.  This  technique,  called  the 
Wald-Wolfowitz  runs  test,  looks  at  the  order  or  sequence  in  which 
the  original  scores  or  measures  were  obtained  (Siegel,  1956).  It 
examines  order  only  and  attempts  to  determine  if  it  could  have 
occurred  randomly  or  if  there  was  some  system  or  method  to  it.  A 
significant  test  result  indicates  that  it  is  unlikely  that  events 
are  random.  When  Wald-Wolfowitz  was  applied  to  each  of  the  time 
blocks  on  fixation  frequency,  two  blocks  were  significant:  the 
10-  to  15-minute  block  (2=2.686,  P<.01)  and  the  20-  to  25-minute 
block  (2=2.207,  P<.05).  Using  a  remote  test  such  as  this  could 
be  rightly  construed  as  reaching  very  hard  for  significance  and 
no  general  conclusions  should  be  drawn  concerning  fixation 
frequency  and  the  various  time  blocks. 

Fixation  durations  are  plotted  in  figure  6.  A  three-way  ANOVA 
was  computed  and  results  indicated  no  significant  interactions 
and  no  main  effects  for  skill  or  taskload  (P>.05).  There  was  a 
significant  effect  on  time  (F=5.23,  P<.01).  When  the  first  time 
block  was  removed,  then  the  time  effect  was  no  longer  significant 
(F=1.29,  P>.05),  indicating  that  saccade  durations  increased 
during  the  beginning  of  a  shift  but  not  subsequently  after  the 
first  5  minutes.  There  were  no  differences  within  time  blocks 
using  Wald-Wolfowitz  or  any  other  statistical  technique. 

Fixation  durations  increased  significantly  during  the  first  5 
minutes  or  so  of  a  shift,  then  stabilized  for  the  remainder  of 
the  scanning  period. 

SCAN  PLOT  QUALITATIVE  ANALYSIS. 

The  oculometer  used  in  this  experiment  provided  the  opportunity 
to  plot  the  results  of  each  simulation  in  terms  of  fixation 
frequencies  and  magnitudes  overlaid  on  a  relative  frame  of 
reference.  This  means  that  while  the  system  did  not  have  the 
capability  to  provide  specific  point  of  gaze  in'^ormatioh  such  as 
what  aircraft  a  controller  was  scanning,  it  co»ild  provide  the 
pattern  of  fixations  relative  to  each  other.  This  pattern, 
coupled  with  a  knowledge  of  the  airspace  geometry,  provides  a 
qualitative  indicator  of  how  controllers  scanned  over  time. 
Further,  the  pattern  and  the  frequency  of  longer  duration 
fixations  indicated  graphically  by  larger  circles  depicting 
fixation  points  can  be  compared  rather  easily  to  the  data  just 
discussed  above. 

It  was  decided  to  plot  each  set  of  oculometer  data  in  5  minute 
blocks  so  1/2  hour  simulation  consisted  of  six  plots  presented  on 
two  pages.  Since  each  of  the  ten  participants  had  two  1/2  hour 
simulations  with  the  oculometer,  the  total  number  of  plots  was 
20  X  6  or  120  in  all.  Rather  than  present  them  here,  they  are 
provided  in  total  in  appendix  C.  They  are  organized  in  two 
groups  based  on  the  participants  skill  level.  Each  set  of  plots 
is  presented  in  the  order  of  the  simulation  runs  regardless  of 
the  order  of  task  loading  which,  for  the  most  part,  did  not  make 
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much  difference  anyway.  The  reader  may  wish  to  review  these 
plots  before  or  during  the  following  brief  qualitative  summary  of 
what  the  plots  may  mean. 

It  is  known  from  the  previous  analysis  that  fixation  duration 
increases  over  the  first  minutes  or  so,  or  the  scanning  period. 
This  would  be  depicted  graphically  as  an  increase  in  the  number 
of  larger  circles  as  one  moves  from  the  first  plot  within  each 
run  to  the  second  and  occasionally  to  the  third.  By  the  third 
plot,  fixation  durations  are  relatively  stable.  The  plots 
support  the  previous  numerical  analysis.  While  there  was  no 
measure  of  point  of  gaze,  the  patterns  of  the  plots  and  the 
changes  over  time  blocks  were  informative.  The  controllers  were 
working  the  southern  approaches  and  were  vectoring  aircraft  to  a 
final  approach  from  west  to  east  on  a  right  parallel  runway.  The 
fixation  patterns  indicate  a  varied  search  over  the  first  5 
minutes  or  so  then  a  concentration  over  time  as  the  traffic 
pattern  developed.  Particularly  for  FPL  controllers,  the  largest 
clusters  of  fixations  were  on  and  around  the  turn  point  from  base 
leg  to  final  approach.  There  was  more  variability  cunong  the 
developmentals,  some  of  whom  did  not  concentrate  as  consistently 
on  the  turn  point.  It  is  not  possible  to  determine  from  these 
plots  what  the  relative  order  of  fixations  were  and  the  magnitude 
of  the  saccades  between  the  fixations. 

The  plots  confirm  that  it  takes  5  to  10  minutes  for  the  scan 
pattern  to  stabilize  when  the  traffic  flow  is  building  and  the 
controller  enters  a  cold  display.  This  was  induced,  in  part, 
because  the  traffic  pattern  and  flow  had  to  build  over  time  as 
simulated  aircraft  entered.  It  is  likely  that  it  would  still 
take  a  few  minutes  for  the  controller  coming  on  to  a  position  to 
develop  his/her  situation  awareness  and  identify  the  best 
scanning  strategy  for  the  situation.  Ideally,  this  should  happen 
prior  to  relieving  the  other  controller.  If  the  new  controller 
says  "I've  got  it"  too  soon,  he  may  well  miss  critical 
information  because  he  is  not  adequately  scanning  the  airspace. 
The  plots  support  a  hypothesis  that,  at  least  in  the  situation 
described  by  the  simulation,  airspace  geometry  and  traffic  flow 
are  major  determiners  of  scan  pattern. 

PERSONALITY  AND  AIR  TRAFFIC  CONTROL. 


Every  air  traffic  controller  is  tested  extensively  when  they 
enter  the  profession.  One  of  the  tests  they  take  is  called  the 
"Sixteen  Personality  Factor  Questionnaire  (16  PF)."  This  is  a 
trait  based  measure  which  attempts  to  identify  where  people  are 
on  16  dimensions  of  personality  that  were  developed  over  40  years 
ago  based  on  factor  analyses  of  all  the  personality  trait 
inventories  available  at  the  time.  The  16  factors  are  listed  in 
table  19. 
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The  16  PF  has  evolved  into  one  of  the  most  heavily  used  and 
researched  personality  questionnaires  in  the  history  of 
psychological  measurement.  The  author  of  this  report  has  used  it 
successfully  in  previous  research,  and  it  was  decided  to 
administer  the  questionnaire  to  the  participants  to  see  if  any 
differences  existed  between  the  developmental  and  FPL  controllers 
(Stein  and  Meiselman,  1977). 

They  are  designated  by  letters  with  the  exception  of  the  last 
four  which  have  letters  and  numbers.  As  indicated  earlier  in  the 
description  of  factor  analysis,  factors  are  usually  named  based 
on  the  variables  that  load  on  them.  The  authors  of  the  test  in 
the  version  used  for  this  experiment  were  very  creative  and 
somewhat  pedantic  when  they  named  the  factors  (Cattell,  Eber,  and 
Tatsuoka,  1970).  For  example.  Factor  "A"  is  referred  to  as  the 
Sizothymia-Af fectothymia  factor.  Since  such  words  do  not  have 
much  meaning  to  the  general  public  or  to  many  psychologists  any 
more,  words  in  more  common  usage  were  selected  to  help  explain 
the  results. 


TABLE  19.  DEFINITIONS  OF  16  PF  FACTORS 


Factor 

A 

B 

C 

D 

E 

F 

G 

H 

I 

L 

M 

N 

O 

Q1 

Q2 

Q3 

Q4 


Low  Score 
Reserved 

Less  Intelligent 

Emotional 

Undemonstrative 

Dependent 

Slow,  Cautious 

Frivolous 

Shy 

Self  Reliant 

Trusting 

Practical 

Unpretentious 

Self  Assured 

Conservative 

Follower 

Uncontrolled 

Relaxed 


High  Score 
Outgoing 

More  Intelligent 
Calm 

Overactive 

Independent 

Quick,  Alert 

Responsible 

Adventurous 

Insecure 

Suspicious 

Imaginative 

Worldly 

Apprehensive 

Experimenting 

Leader 

Controlled 

Tense 


The  16  PF  consists  of  a  series  of  questions  in  which  the 
respondent  is  asked  to  indicate  the  degree  to  which  certain 
statements  like  "I  like  to  watch  team  games"  apply  to  him.  The 
questionnaire  is  scored  for  a  raw  score  which  is  compared  against 
a  standardization  sample,  the  results  of  administering  the 
questionnaire  to  many  people.  The  result  is  a  Standard  Ten  Score 
(STEN)  score  on  each  of  the  16  factors.  A  STEN  relates  the 
individuals  raw  score  against  the  performance  of  the  group  in  the 
standardization  sample.  The  mean  of  the  distribution  of  STEN 
scores  is  5.5.  The  range  from  5  to  6  covers  one  standard 
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deviation  and  the  authors  suggest  that  only  steps  that  are  at  or 
below  4  or  equal  or  exceed  7  should  be  considered  as  having 
departed  from  the  average. 

Using  this  inloiination  helps  interpret  figure  1,  which  maps  the 
mean  results  for  the  ten  controller  participants  on  the  16  PF. 
This  figure  indicates  that  there  may  have  well  been  differences 
between  FPL  and  developmental  controllers  at  least  on  some 
dimensions.  Figure  7  indicated  that  there  were  a  number  of 
dimensions  where  one  group  or  the  other  diverged  from  the  average 
and  met  or  exceeded  the  4  or  7  STEN  score  cut  points.  A  low 
score  on  factor  "E"  for  the  developmental  controllers 
demonstrated  that,  on  average,  they  responded  on  the  16  PF  that 
they  felt  dependent  and  humble.  This  was  not  the  case  for  the 
more  confident  FPL's.  A  very  low  score  on  factor  ”M”  by  the 
developmental  controllers  indicated  a  focus  on  practical  concerns 
and,  according  the  16  PF  handbook,  that  they  may  have  felt 
concerned  or  worried  when  they  completed  the  questionnaire.  A 
high  score  on  Q2  for  the  FPL's  indicated  that  they  felt 
considerable  self-sufficiency  and  resourcefulness,  which  is 
characteristic  of  controllers  as  a  whole.  The  developmentals 
also  were  about  a  half  a  standard  deviation  above  the  mean  on 
this  variable. 

In  order  to  determine  whether  there  were  any  differences  between 
the  two  groups  of  controllers  that  occurred  beyond  the  level  of 
chance,  a  very  basic  statistical  procedure  was  used.  A  "t"  test 
was  computed  between  the  two  groups  on  each  of  the  16  factors. 

The  "t"  test  is  like  a  subset  of  analysis  of  variance.  It  allows 
for  a  probability  estimate  that  difference  between  two  means  may 
have  occurred  beyond  a  chance  level.  The  results  for  the  "t" 
tests  computed  are  shown  in  table  20. 

Only  those  t's  that  were  significant  from  zero  (P<.01)  are 
reported  in  this  table.  Results  indicated  that  the  FPL's  and 
developmentals  were  significantly  different  on  4  of  the  16 
factors.  The  FPL's  described  themselves  as  more  self-assured 
(E),  more  imaginative  (M)  than  the  developmental  controllers. 

The  developmentals,  who  may  have  been  overcompensating,  saw 
themselves  as  more  worldly/ shrewd  (M)  than  did  the  FPL's,  but 
admitted  that  they  were  more  apprehensive  (O)  .  tfhile  the 
differences  between  the  two  groups  were  not  significant  on  factor 
Q2,  the  FPL's  did  exceed  the  cut  point  of  7,  Indicating  that  they 
felt  very  self-assured  and  resourceful. 

These  data  are  more  relevant  when  it  is  noted  that  the  16  PF  was 
not  administered  until  the  very  end  of  the  experiment,  after  all 
the  simulations  were  completed.  This  suggests  that  the  values 
achieved  represented  more  than  a  response  to  the  immediate 
situation  and  may  have  tapped  more  enduring  characteristics  of 
the  ten  individuals  who  participated  in  the  study. 
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TABLE  20.  PERSONALITY  DIFFERENCES  BETWEEN  CONTROLLER 
GROUPS 


Personality  Factor  t  Value 


E  { Dependent- Independent )  3.93 
M  (Practical-Imaginative)  4.72 
N  (Unpretentious-Worldly)  4.67 
O  (Self  assured-Apprehensive )  3.49 


The  results  of  the  post-run  attitude  questionnaires  which  were 
administered  after  every  simulation  were  described  previously  in 
this  report.  Given  that  personality  is,  in  theory,  more  enduring 
than  attitude  and  opinion  measured  by  the  questionnaires,  the 
relationship  between  the  16  PF  and  questionnaire  responses  was 
examined.  Pearson  product-moment  correlations  were  computed 
between  the  16  factors  and  the  questionnaire  data,  to  which  was 
added  the  observers'  performance,  workload  ratings,  and  the 
number  of  landings  achieved  by  the  participants.  Table  21 
describes  these  correlations. 


TABLE  21.  PERSONALITY  POST-RUN  QUESTIONNAIRE  CORRELATIONS 


Questionnaire 

16  PF 

FACTORS 

ft  Perforaance 

A 

B 

C 

E 

£ 

G 

H 

1 

L 

N 

N 

0 

91 

9i 

9^ 

04 

Skill 

-.11 

.21 

-.12 

-.54 

-.23 

.29 

-.30 

.06 

.07 

-.61 

.60 

.49 

.11 

-.24 

.20 

-.10 

Uorkload 

-.12 

.25 

-.37 

.14 

-.22 

-.15 

-.45 

.14 

.48 

.16 

-.11 

.25 

-.09 

.39 

-.54 

.40 

Perfora 

-.05 

-.49 

.59 

.58 

.63 

.10 

.51 

-.42 

.07 

.16 

-.68 

-.55 

.36 

-.27 

.40 

-.23 

Tiaecont 

-.05 

.05 

-.23 

.27 

-.07 

.05 

-.42 

-.03 

.50 

.12 

-.09 

.06 

.31 

-.44 

.12 

Stress 

.04 

.38 

-.50 

-.21 

-.27 

.09 

-.53 

.12 

.36 

-.13 

.17 

.26 

.07 

.17 

-.17 

.18 

Interfer 

.21 

.26 

-.26 

-.03 

-.05 

.19 

-.27 

.13  - 

.04 

.02 

-.07 

-.14 

-.24 

-.08 

-.02 

-.13 

Trafvolu 

-.20 

.04 

.09 

.48 

.01 

.00 

-.27 

.10 

.55 

.22 

-.22 

.18 

-.21 

.15 

-.52 

.03 

Trafcoap 

-.23 

.16 

-.05 

.20 

-.15 

-.05 

-.35 

.14 

.25 

.09 

.04 

.33 

-.34 

.23 

-.62 

.24 

Ry layout 

-.44 

-.32 

.40 

.31 

.26 

.19 

-.14 

-.13 

.14 

-.02 

-.42 

.06 

.33 

-.19 

.06 

-.34 

ACedges 

-.34 

-.20 

.42 

.30 

.38 

.29 

-.01 

-.17 

.30 

-.26 

-.43 

-.04 

.08 

-.39 

.17 

-.10 

Metering 

-.40 

-.35 

.60 

.71 

.42 

.07 

.01 

-.28 

.09 

.31 

-.61 

-.18 

.16 

-.01 

-.23 

-.19 

Fatigue 

.10 

.07 

.08 

.06 

.12 

.42 

-.27 

-.18  - 

.12 

-.07 

.08 

-.07 

.20 

-.18 

.16 

-.51 

LAND 

.32 

.16 

.02 

-.17 

-.01 

.42 

-.01 

.02 

.11 

-.37 

.51 

.12 

-.19 

-.36 

.28 

-.42 

WLRTG 

.10 

-.18 

.08 

.19 

.21 

.28 

.07 

-.19 

.25 

-.37 

.08 

.31 

-.37 

-.12 

-.10 

.17 

PERFRT 

.00 

-.15 

.51 

.32 

.33 

-.02 

.44 

-.05  - 

.20 

.26 

-.41 

-.63 

.00 

-.32 

.31 

-.33 

Mote:  Underlined  correlations  are  considered  significant  froa  zero. 


In  order  to  approach  these  data  conservatively,  it  was  decided  to  use 
the  P<.01  level  of  significance  to  interpret  the  correlations.  The 
critical  value  for  correlation  using  this  level  of  significance  is 
r=.53.  Only  those  which  met  or  exceeded  this  value  were  considered  as 
significant  from  zero  beyond  a  chance  level.  Out  of  240  correlations, 
only  15  were  significant,  and  these  are  underlined  in  the  table. 

The  significant  correlations  indicate  a  relationship  between  current 
responses  to  the  immediate  environment  and  what  the  controllers 
brought  with  them  to  the  situation.  In  terms  of  skill  level,  factors 
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E,  M,  and  N  were  significantly  correlated.  This  was  not  surprising 
since  it  was  already  shown  in  an  earlier  analysis  that  these  factors 
separated  the  two  groups  of  participants.  There  was  one  16  PF 
variable,  Q3  (Uncontrolled-Controlled)  that  was  related  to  self- 
assessment  of  workload  in  the  questionnaire.  Those  who  defined 
themselves  as  more  uncontrolled  and  spontaneous  rated  their  workload 
during  the  simulations  as  higher.  Controllers'  self-assessment  of 
performance  was  significantly  related  to  factors  "C"  (Emotional-Calm), 
"E”  ( Dependent- I ndependent ) ,  "F"  (Slow/Cautious-QuickMlert ) ,  ”N” 
(Unpretentious-Worldly),  and  ”0"  ( Self -Assured-Apprehensive ) .  Those 
who  rated  their  own  performance  as  higher  saw  themselves  as  more  calm, 
independent,  quick/alert,  unpretentious,  and  self-assured.  There  were 
no  significant  relationships  for  the  third  questionnaire  item,  the 
fraction  of  time  actually  controlling  traffic  (Timecont).  The  stress 
question  was  inversely  related  to  factor  "H"  (Shy-Adventuresome). 

Those  who  had  less  stress  in  the  simulations  scored  more  adventurous 
on  the  16  PF. 

There  were  no  significant  correlations  with  the  degree  to  which  the 
participant  felt  the  oculometer  interfered  with  his  performance. 
Controllers'  perceptions  of  how  traffic  volume  influenced  their 
scanning  were  correlated  with  factor  "L. ”  Those  who  indicated  that 
they  were  more  trusting  on  the  16  PF  saw  less  of  an  impact  by  traffic 
volume.  The  observed  impact  of  traffic  complexity  was  related  to 
factor  Q3  (Uncontrolled-Controlled).  Participants  saw  a  greater 
impact  of  traffic  complexity  if  they  scored  towards  the  uncontrolled 
end  of  factor  ”L."  Runway  layout  and  aircraft  at  the  edges  of  the 
display  were  not  related  to  16  PF  data.  The  impact  of  metering  on 
scanning  was  significantly  correlated  with  factor  "C"  (Emotional- 
Calm),  "E"  ( Dependent- Independent )  and  "N”  (Unpretentious-Worldly). 
Controllers  who  saw  metering  as  more  important  for  their  scanning  were 
more  calm,  independent,  and  unpretentious  on  the  16  PF.  There  were  no 
relationships  of  any  magnitude  between  the  16  PF  and  the  fatigue 
question  on  the  questionnaire,  the  number  of  a  landings  the  controller 
achieved,  or  the  workload  rating  that  the  observer  recorded.  There 
was,  however,  an  inverse  correlation  between  the  observer's 
performance  rating  and  factor  "O"  (Self-Assured-Apprehensive ) .  Higher 
performance  ratings  were  given  to  those  who  were  more  self-assured  on 
the  16  PF.  This  last  finding  is  probably,  at  least  in  part,  a 
function  of  the  fact  that  the  FPL's  received  higher  performance 
ratings  and  were  more  also  self-assured.  The  observer  may  have  been 
relating  in  part  to  the  FPL's  confidence  in  their  own  skills. 

Results  of  the  correlations  of  16  PF  data  with  post-run  questionnaires 
indicate  that  controllers  respond  to  the  immediate  situation  based,  in 
part,  on  what  they  brought  with  them  to  that  environment.  Personality 
is  one  aspect  of  who  we  are,  and  it  is  related  to  how  controllers  view 
their  surroundings. 
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EXIT  INTERVIEWS. 


At  the  end  of  the  experiment,  when  all  the  simulations  were  completed, 
each  of  the  participants  was  interviewed  concerning  his  experiences 
and  opinions.  Each  was  asked  to  rate  the  realism  of  the  simulation  on 
a  10-point  scale  from  l=low  to  10=high.  The  range  of  responses  was 
from  3  to  10  with  a  median  of  7 .  This  is  a  reasonable  response  in 
comparison  with  previous  studies  using  the  NSSF  simulation  at  the 
Technical  Center.  When  asked,  what  aspects  of  the  simulation  were 
different  from  their  home  facility,  everyone  found  something  that  was 
different.  This  included  the  fact  that  in  the  experiment  flight 
strips  were  not  used,  and  that  the  equipment  in  the  NSSF  was  a  little 
different  in  appearance  from  what  they  were  used  to  operationally. 

They  also  noted  that  ordinarily  they  would  have  separate  feeder  and 
final  approach  positions.  They  were  asked  to  do  both,  basically 
because  there  were  not  enough  controllers.  However,  the  task  load  was 
adjusted  so  that  no  one  was  overloaded  during  the  experiment.  The 
controller  participants  also  noted  that  the  aircraft  performed  at 
standard  rates  and  more  consistently  than  they  do  in  the  real  world. 
This  is  an  old  problem  with  the  simulation,  which  may  be  rectified  by 
newer  editions  of  the  target  generation  functions  and  hardware. 

Controllers  were  asked  about  their  perceptions  of  the  impact  of  the 
oculometer  on  their  ability  to  control  traffic.  The  responses  ranged 
from  constant  awareness  of  the  presence  of  the  oculometer  with  some 
resultant  annoyance,  particularly  from  the  helmet  and  the  visor  to  the 
ability  to  tune  out  the  equipment  completely.  This  ability  to  ignore 
the  equipment  was  based  on  distraction  by  the  traffic  and  control 
requirements.  Controllers  indicated  that  as  they  became  busier  they 
noticed  the  oculometer  less.  Out  of  ten  controllers,  seven  saw  no 
oculometer  impact  at  all  on  their  performance.  Two  indicated  that  it 
bothered  them  only  at  the  beginning  of  the  experiment.  One 
participant  felt  more  tense  when  using  the  oculometer.  He  said  he 
felt  "under  the  gun." 

When  asked  whether  they  had  a  specific  strategy  for  scanning  the 
planned  view  display,  most  controllers  had  to  think  awhile  in  order  to 
review  what  it  was  they  were  actually  doing.  Almost  all  indicated 
that  their  scanning  methods  were  situation  dependent.  Factors  that 
they  considered  included:  traffic  volume  and  flow,  the  geometry  of 
the  airspace  at  the  position  being  worked,  aircraft  speed 
compatibility,  weather,  flow  control,  and  critical  points.  Verbalized 
strategies  were  in  line  with  the  scan  plot  qualitative  analysis 
reported  earlier.  Controllers  indicated  that  they  focused  on  critical 
turn  points  such  as  downwind  to  base  and  base  to  final  legs  of  the 
approach. 

Controllers  were  asked  if  they  had  a  characteristic  style  and 
preferences  concerning  separation  techniques.  Of  specific  interest 
was  whether  they  preferred  to  use  longitudinal  or  vertical  separation. 
This  question  was  not  asked  because  its  impact  on  visual  scanning; 
rather,  the  goal  was  to  determine  whether  the  participants  were 
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thinking  like  those  in  previous  studies,  and  also  to  build  an  ongoing 
body  of  knowledge  on  controller  behavior.  Results  indicated  that 
while  the  majority  of  controllers  believed  that  separation  strategy 
should  be  tailored  to  the  tactical  situation,  other  things  being 
equal,  they  preferred  to  use  vertical  separation  techniques.  This 
reduces  workload  because  the  controller  did  not  have  to  keep 
estimating  the  distance  between  aircraft  which  is  required  if  they  are 
coaltitude.  This  result  is  consistent  with  previous  research  done  at 
the  FAA  Technical  Center  in  which  the  focus  was  parallel  approach 
separation  (Stein,  1989b). 

Finally,  controllers  were  asked  if  there  was  anything  that  should  have 
been  asked  that  was  not  or  if  there  was  anything  they  felt  the 
experimenter  should  know.  One  controller  expressed  his  displeasure 
with  the  oculometer  helmet.  A  few  again  noted  the  differences  between 
the  simulation  and  their  home  facility,  but  did  not  have  any  major 
concerns  with  them.  One  indicated  the  equipment  was  fine,  the 
experiment  was  well  prepared,  and  the  practice  runs  prior  to  data 
collection  were  useful.  One  controller  commented  that  the  FPL 
personnel  would  have  performed  even  better  if  they  had  flight  strips 
which  he  uses  for  planning  purposes.  Another  controller  suggested 
that  more  mistakes  are  made  in  ATC  when  they  are  not  busy  because 
their  thinking  wanders  and  it  is  easier  to  be  distracted.  While  this 
did  not  relate  directly  to  the  scanning,  it  is  in  line  with  FAA  error 
analyses.  One  of  the  developmental  controllers,  who  had  been  a 
controller  at  another  facility,  said  that  the  simulation  of  his 
current  facility  was  a  learning  experience  for  him  because  he  had  not 
yet  been  radar  qualified  in  his  new  job. 

The  controller  visual  scanning  study  was  completed  and  the  results 
indicated  that  their  was  form  and  substance  to  the  way  controllers 
reach  out  for  information.  In  their  own  view,  controllers  each 
develop  a  style  that  works  for  them  and  they  attempt  to  make  the  best 
use  of  it  even  though  they  may  not  be  continually  aware  of  what  it  is 
they  are  doing  visually.  When  exposed  to  the  quasi  realistic 
environment  of  the  simulations,  they  attempted  to  work  with  the 
equipment  and  live  with  the  inconvenience  of  the  oculometer.  As 
always,  controllers  are  adaptable  and  they  work  with  what  they  have. 
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APPENDIX  A 


TRAINING/FAMILIARIZATION  PLAN 


TRAINING/FAMILIARIZATION  PLAN 


PRIMARY  OBJECTIVE. 

Familiarize  the  participant  controllers  with  the  Air  Traffic 
Control  Simulation  Facility  and  the  airspace  geometry  which  it 
simulates.  Ensure  that  participants  are  able  to  control  a  moderate 
level  of  traffic,  using  the  appropriate  procedures  and  techniques. 

ENABLING  OBJECTIVES. 

1.  Condition;  Given  a  routine  air  traffic  sample  of  ten  or  less 
aircraft  in  sector. 

Task;  The  participant  maintains  communications  with  aircraft 
under  his/her  control  and  with  adjacent  controllers  as  required 
for  intersector  coordination. 

Standard;  The  participant  employs  standard  radio  telephone 
procedure,  initiates  contact  to  obtain  required  information  or 
provide  information  and  directives,  accomplishes  all  necessary 
land  line  coordination  with  adjacent  sectors. 

2.  Condition;  Given  a  briefing  and  documents  concerning 
operational  procedures. 

Task;  The  participant  demonstrates  his/her  knowledge  and 
acceptance  of  these  procedures  through  verbal  discussion  with 
the  training  controller. 

Standard;  The  training  controller  verifies  that  the 
participant  has  a  working  knowledge  of  procedures. 

3.  Condition;  Given  air  traffic  sample  of  ten  or  less  aircraft 
of  mixed  types  and  flightpaths  where  potential  conflicts,  are 
preprogrammed . 

Task;  The  participant  maintains  radar  surveillance, 
anticipates  and  identifies  potential  conflicts,  and  issues 
amended  clearances. 

Standard ;  During  a  1-hour  simulation  the  participant 
controller  does  not  allow  more  than  two  violations  of  the 
horizontal  separation  standard  of  aircraft  within  the  vertical 
separation  envelope  and  in  no  case  are  the  violations  allowed 
to  progress  to  a  point  closer  than  2  miles  of  separation. 

4.  Condition;  Given  an  air  traffic  sample  of  15  or  fewer  aircraft 
of  mixed  types  and  flightpaths  where  conflicts  of  separation 
may  or  may  not  occur. 
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Task :  The  participant  exercises  traffic  management  techniques 
to  minimize  delays  and  maintain  a  positive  and  expeditious 
traffic  flow. 

Standard ;  The  controller  maintains  positive  command  of  the 
traffic  flow  and  introduces  path  changes  only  where  necessary 
to  maintain  safe  efficient  traffic  flow. 

5.  Condition ;  Given  this  training/familiarization  program 

involving  briefings,  printed  material  and  "hands-on"  control 
of  simulated  aircraft. 

Task;  The  participant  controller  is  able  to  control  traffic. 

Standard ;  The  participant  is  willing  to  state  that  he/she  is 
adequately  familiar  with  the  simulation  so  that  the  simulation 
itself  does  not  inhibit  his/her  performance. 
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APPENDIX  B 


FORMS  AND  QUESTIONNAIRES 


CONTROLLER  SIMULATION  QUESTIONNAIRE 


INSTRUCTIONS 


PLEASE  COMPLETE  THE  FOLLOWING  QUESTIONS  AS  SOON  AS  YOU  HAVE  BEEN 
RELIEVED  FROM  YOUR  RADAR  POSITION.  YOUR  RESPONSES  SHOULD  FOCUS  ON 
ONLY  THE  WORK  THAT  YOU  HAVE  JUST  COMPLETED  IN  THE  LAST  CONTROL 
PERIOD. 

ALL  CONTROLLERS  EXPERIENCE  A  WIDE  VARIETY  OF  ACTIVITY  AND  RESULTANT 
WORKLOAD  DURING  THEIR  CAREERS.  IT  DOES  NOT  DETRACT  FROM  YOUR 
PROFESSIONALISM  IF  FOR  A  GIVEN  PERIOD  YOU  REPORT  VERY  HIGH  OR  VERY 
LOW  WORKLOAD.  ON  ALL  THE  QUESTIONS  WHICH  FOLLOW  FEEL  FREE  TO  USE 
THE  ENTIRE  NUMERICAL  SCALE  FOR  EACH  ANSWER.  BE  AS  HONEST  AND  AS 
ACCURATE  AS  YOU  CAN.  YOUR  NAME  IS  MOT  RECORDED  ON  THIS  OR  AMY  OTHER 
FORM,  AND  MO  ATTEMPT  WILL  BE  MADE  TO  ASSOCIATE  YOUR  RESPONSES  WITH 
YOU  AS  AN  INDIVIDUAL.  DATA  COLLECTED  HILL  BE  FOR  RESEARCH  PURPOSES 
ONLY.  THANK  YOU  FOR  YOUR  PARTICIPATION. 
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POST  RDM  CONTROLLER  QUESTIONNAIRE 


PARTICIPANT  CODE 


DATE 


RUN  NUMBER 


RUNWAY 


(Circle  One) 

OCCULOMETER  yes 


no 


TIME 


1.  CHOOSE  THE  ONE  NUMBER  BELOW  WHICH  BEST  DESCRIBES  HOW  HARD 
YOU  WERE  WORKING  DURING  THIS  PERIOD: 


DESCRIPTION  OP  WORKLOAD  RATING 

_  CATEGORY _  f CIRCLE  ONE^ 

VERY  LOW  WORKLOAD-  ALL  TASKS  WERE  1 

2 

WERE  ACCOMPLISHED  EASILY  &  QUICKLY  3 


MODERATE  WORKLOAD-  THE  CHANCES  FOR  4 

5 

ERROR  OR  OMISSION  WERE  LOW  6 


RELATIVELY  HIGH  WORKLOAD-  THE  CHANCES  7 
FOR  SOME  ERROR  OR  OMISSION  WERE  8 
RELATIVELY  HIGH  9 


VERY  HIGH  WORKLOAD  -  IT  WAS  10 
BARELY  POSSIBLE  TO  ACCOMPLISH  11 
ALL  TASKS  PROPERLY  12 


2. RATE  YOUR  PERFORMANCE  CONTROLLING  TRAFFIC  DURING  THE  PAST 
HOUR.  CIRCLE  THE  NUMBER  WHICH  BEST  DESCRIBES  HOW  WELL  YOU 
THINK  YOU  DID. 

123456789  10 

AVERAGE  EXCELLENT 


3.  WHAT  FRACTION  OF  THE  TIME  WERE  YOU  BUSY  DURING  THE  PERIOD 
YOU  WERE  CONTROLLING? 

123456789  10 

SELDOM  HAD  FULLY  OCCUPIED 

MUCH  TO  DO  AT  ALL  TIMES 
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4.  RATE  THE  DEGREE  TO  WHICH  YOU  FOUND  THIS  CONTROL  PERIOD 
STRESSFUL!  CIRCLE  THE  NUMBER  BELOW  WHICH  BEST  DESCRIBES 
HOW  YOU  FELT. 

123456789  10 

LOW  HIGH 

STRESS  STRESS 


5.  PLEASE  RATE  YOUR  LEVEL  OF  AGREEMENT  WITH  FOLLOWING 

STATEMENT  REGARDLESS  OF  WHETHER  YOU  USED  THE  OCCULOMETER 
DURING  THIS  LAST  CONTROL  PERIOD. 


The  occulometer  did  not  interfer  in  any  way  with  my 
performance  during  this  control  period! 

12345678 
Strongly 
Agree 


9  10 

Strongly 

Disagree 


6.  Below  circle  the  number  from  l>lov  to  10-high  which  best 
describes  the  influence  of  each  of  the  following  factors 
on  how  you  visually  scanned  your  radar  display  during  the 
last  period. 


Traffic  Volume 

LOW 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

High 

Traffic 

Composition 

Low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

High 

Runway  layout 

Low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

High 

Aircraft  at  the 
display  edges 

Low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

High 

Metering  of 
inbound  traffic 

Low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

High 

Personal 

Fatigue 

Low 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

High 

7.  Briefly  describe  your  strategy  for  working  traffic  during 
this  control  period. 
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8.  IS  THERE  ANYTHING  ELSE  THAT  HAPPENED  THIS  PAST  HOUR  WHICH 
YOU  FEEL  MIGHT  HELP  US  UNDERSTAND  THE  RESULTS?  ANY 
COMMENTS  YOU  HAVE  AT  THIS  POINT  WOULD  BE  VERY  WELCOME. 


FAA  TECHNICAL  CENTER 
CONTROLLER  ENTRY  QUESTIONNAIRE 


The  purpose  of  this  questionnaire  is  to  find  out  something 
about  your  background  and  current  feelings  about  this  project 
in  order  to  better  understand  your  performance  during  the 
course  of  the  study.  All  information  is  collected  under  your 
code  number  and  no  attempt  will  be  made  to  link  your  name  to 
to  the  answers  you  provide.  Welcome  to  the  Technical  Center 
and  thank  you  for  your  participation! 


1 . Participant  Code: 


Todays  Date: 


2. Your  total  experience  as  a  Controller  _ 

years  months 


3.  Your  experience  at  your  current  facility  _ 

years  months 

4.  Your  current  position:  FPL,  Developmental,  other _ 

5.  Please  rate  your  current  vision  on  the  scale  below: 

123456789  10 

Poor  Excellent 


Please  indicate  your  level  of  agreement  with  each  of 
the  following  statements  by  circling  the  most  appropriate 
number  between  1  Strongly  Disagree  and  10  Strongly  Agree. 


6.  I  freely  volunteered  to  participate  in  this  project. 


12345678 

Strongly 

Disagree 

9 

10 

Strongly 

Agree 

7.  I  currently  am  in  good  health. 

12345678 

Strongly 

Disagree 

9 

10 

Strongly 

Agree 

8. During  the  last  several  months,  I  have  been 
a  relatively  high  level  of  stress. 

experiencing 

12345678 

Strongly 

Disagree 

9 

10 

Strongly 

Agree 

9 .  I  am  not  very  motivated  to  participatate  in 

this  study. 

123456  78 

Strongly 

Agree 

9 

10 

Strongly 

Disagree 
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OBSERVER  EVALUATION  FORM 


RUN  NO  _  DATE 

PARTICIPANT 

OBSERVED  TIME 


1 .  BELOW  PLEASE  CIRCLE  THE  NUMBER  WHICH  BEST  DESCRIBES  HOW 
HARD  THE  CONTROLLER  WAS  WORKING  DURING  EACH  15  MINUTE 
BLOCK  OF  THIS  RUN. 


FIRST  BLOCK 

1 

2 

4 

5 

6 

7 

8 
9 

10 


VERY  EASY 


VERY  HARD 


SECOND  BLOCK 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 


: .  BELOW  PLEASE  CIRCLE  THE  NUMBER  WHICH  BEST  DESCRIBES  CONTROLLER 
EFFECTIVENESS  DURING  EACH  BLOCK  OF  THIS  RUN. 


FIRST  BLOCK 


SECOND  BLOCK 


1  AVERAGE  1 

2  2 

3  3 

4  4 

5  5 

6  6 

7  7 

8  8 

9  9 

10  EXCELLENT  10 


In  this  space  please  count  or  tally  the  number  of  times  observed 
controller  exhibits  behaviors  which  are  not  related  directly  to 
control  duties (ie  talks  to  observers,  looks  away  from  the  display) 

First  Block  Second  Block 


Place  any  comments  or  other  observations  on  the  back  of  form. 
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CONTROLLER  VISUAL  SCANNING 
EXIT  INTERVIEW 


DATE 


PARTICIPANT 


1. Please  rate  the  realism  of  this  simulation  from  1  low  to  10  high. 


2 .  Was  there  anything  that  you  found  particulai iy  unique  in  the 
simulation  that  you  would  not  see  at  your  home  facility? 


3.  Think  back  to  those  runs  when  you  were  using  the  oculometer. 
Were  you  constantly  aware  of  it  or  did  you  tune  it  out? 

After  you  wore  it  for  a  few  minutes  was  it  annoying? 


Did  the  use  of  the  oculometer  effect  your  performance  in  any 
way? 


4.  Think  about  how  you  search  the  PVD  for  information.  Do  you  do 
it  in  one  special  way  or  does  it  depend  on  certain  factors 
and  if  so  what  are  they? 


5.  If  you  have  a  choice  of  separating  aircraft  vertically  or 
horizontally  which  do  you  prefer  to  do  and  why.? 


6.  How  do  you  decide  whether  or  not  to  suppress  data. 


7 .  Is  there  anything  I  have  not  asked  you  or  you  think  I  should 
know  which  would  help  me  understand  what  went  on  in  this  study? 
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APPENDIX  C 


OCOLOMETER  PLOTS 


PARTICIPANT  01  RUN  1,  0-5  MINS  H  LOAD  9/6/90 


PARTICIPANT  01  RUN  1,  10-15  MINS  H  LOAD  9/6/90 


PARTICIPANT  01  RUN  1,  15-20  MINS  H  LOAD  9/6/90 


PARTICIPANT  01  RUN  1,  25-30  MINS  H  LOAD  9/6/90 


PARTICIPANT  01  RUN  04,  5-10  MINS  L  LOAD  9/6/90 


PARTICIPANT  01  RUN  04,  10-15  MINS  L  LOAD  9/6/90 


PARTICIPANT  01  RUN  04,  16-20  MINS  L  LOAD  0/0/00 


c- 


PARTICIPANT  03  RUN  12,  10-15  MINS  H  LOAD  0/11/90 
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PARTICIPANT  OS  RUN  12,  lB-20  MINS  H  LOAD  0/11/00 


PARTICIPANT  03  RUN  12,  20-25  MINS  H  LOAD  0/11/00 


PARTICIPANT  03  RUN  12,  25-30  MINS  H  LOAD  0/11/00 


PARTICIPANT  03  RUN  13,  5-10  MINS  L  LOAD  0/11/00 


PARTICIPANT  03  RUN  13,  10-15  MINS  L  LOAD  0/11/00 


C- 


PARTICIPANT  OS  RUN  13,  15-20  MINS  L  LOAD  0/11/00 


PARTICIPANT  03  RUN  13,  20-25  MINS  L  LOAD  0/11/00 


PARTICIPANT  03  RUN  13,  25-30  MINS  L  LOAD  0/11/00 


PARTICIPANT  05  RUN  22,  5-10  MINS  L  LOAD  0/13/00 


PARTICIPANT  05  RUN  22,  10-15  MINS  L  LOAD  0/13/00 
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PARTICIPANT  05  RUN  22,  20-25  MINS  L  LOAD  0/13/90 


C-10 


•PARTICIPANT  05  RUN  23,  5-10  MINS  H  LOAD  9/13/90 


PARTICIPANT  05  RUN  23,  10-15  MINS  H  LOAD  9/13/90 
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PARTICIPANT  05  RUN  23,  15-20  MINS  H  LOAD  9/13/90 


C-12 


PARTICIPANT  07  RUN  31,  0-5  MINS  L  LOAD  1/8/01 


PARTICIPANT  07  RUN  31,  5-10  MINS  L  LOAD  1/8/91 


PARTICIPANT  07  RUN  31,  10-15  MINS  L  LOAD  1/8/91 


PARTICIPANT  07  RUN  31,  15-20  MINS  L  LOAD  1/8/81 


PARTICIPANT  07  RUN  34,  15-20  MINS  H  LOAD  1/8/91 


PARTICIPANT  07  RUN  34,  20-25  MINS  H  LOAD  1/8/91 


PARTICIPANT  07  RUN  34,  25-30  MINS  H  LOAD  1/8/91 


C-16 


PARTICIPANT  07  RUN  34,  5-10  MINS  H  LOAD  1/8/91 


PARTICIPANT  07  RUN  34,  10-15  MINS  H  LOAD  1/8/91 


C-15 


PARTICIPANT  08  RUN  32,  0-5  MINS  H  LOAD  1/8/91 


PARTICIPANT  08  RUN  32,  5-10  MINS  H  LOAD  1/8/91 


PARTICIPANT  08  RUN  32,  10-15  MINS  H  LOAD  1/8/91 


PARTICIPANT  08  RUN  32,  15-20  MINS  H  LOAD  1/8/91 


PARTICIPANT  08  RUN  32,  25-30  MINS  H  LOAD  1/8/91 


PARTICIPANT  08  RUN  33,  15-20  MINS  L  LOAD  1/8/91 


PARTICIPANT  08  RUN  33,  25-30  MINS  L  LOAD  1/8/91 


PARTICIPANT  08  RUN  33,  5-10  MINS  L  LOAD  1/8/91 


PARTICIPANT  08  RUN  33,  10-15  MINS  L  LOAD  1/8/91 


C-20 


PARTICIPANT  10  RUN  42,  10-15  MINS  L  LOAD  1/10/81 


PARTICIPANT  10  RUN  42,  15-20  MINS  L  LOAD  1/10/91 


PARTICIPANT  10  RUN  42,  20-25  MINS  L  LOAD  1/10/91 


PARTICIPANT  10  RUN  42,  25-30  MINS  L  LOAD  1/10/91 


PARTICIPANT  10  RUN  43,  5-10  MINS  H  LOAD  1/10/91 


PARTICIPANT  10  RUN  43,  10-15  MINS  H  LOAD  1/10/91 


C-23 


PARTICIPANT  10  RUN  43,  15-20  MINS  H  LOAD  1/10/91 


o 

PARTICIPANT  10  RUN  43,  20-25  MINS  H  LOAD  1/10/91 


PARTICIPANT  10  RUN  43,  25-30  MINS  H  LOAD  1/10/91 


o 


o 


PARTICIPANT  02  RUN  2R,  0-6  MINS  L  LOAD  0/7/80 


PARTICIPANT  02  RUN  2R,  5-10  MINS  L  LOAD  9/7/80 


PARTICIPANT  02  RUN  2R,  10-15  MINS  L  LOAD  8/7/90 


C-26 


PARTICIPANT  02  RUN  2R,  15-20  MINS  L  LOAD  0/7/00 


PARTICIPANT  02  RUN  2R,  25-30  MINS  L  LOAD  0/7/00 


PARTICIPANT  02  RUN  03,  5-10  MINS  H  LOAD  9/6/90 


PARTICIPANT  02  RUN  03,  10-15  MINS  H  LOAD  9/6/90 


C-28 


ATICIPANT  02  RUN  03,  15-20  MINS  H  LOAD  9/6/00 


iRTICIPANT  02  RUN  03,  20-25  MINS  H  LOAD  9/6/90 


iRTICIPANT  02  RUN  03,  25-30  MINS  H  LOAD  9/6/90 


PARTICIPANT  X>4  RUN  11,  0-5  MINS  L  LOAD  0/11/00 


PARTICIPANT  04  RUN  11,  10-15  MINS  L  LOAD  0/11/00 


ARTICIPANT  04  RUN  11,  16'20  MINS  L  LOAD  9/11/90 


ANTICIPANT  04  RUN  11,  20-25  MINS  L  LOAD  9/11/90 


ANTICIPANT  04  RUN  11,  25-30  MINS  L  LOAD  9/11/90 


PARTICIPANT  04  RUN  14,  0-5  MINS  H  LOAD  9/12/90 


ITICIPANT  04  RUN  14,  15-20  MINS  H  LOAD  O/12/tO 


RTICIPANT  04  RUN  14,  20-25  MINS  H  LOAD  0/12/90 


RTICIPANT  04  RUN  14,  25-30  MINS  H  LOAD  0/12/90 


ARTICIPANT  06  RUN  21,  0-5  MINS  H  LOAD  9/13/80 


ATICIPANT  Oe  RUN  21,  16-20  MINS  H  LOAD  0/13/00 


iRTICIPANT  06  RUN  21,  20-25  MINS  H  LOAD  0/13/00 


RTICIPANT  06  RUN  21,  26-30  MINS  H  LOAD  0/13/00 


C-35 


>ARTICIPANT  06  RUN  24,  0-5  MINS  L  LOAD  9/13/90 


PARTICIPANT  06  RUN  24,  10-15  MINS  L  LOAD  9/13/90 


ARTICIPANT  06  RUN  24,  15-20  MINS  L  LOAD  0/13/00 


ARTICIPANT  06  RUN  24,  25-30  MINS  L  LOAD  0/13/00 


PARTICIPANT  09  RUN  41,  0-5  MINS  H  LOAD  1/10/91 


PARTICIPANT  09  RUN  41,  10-15  MINS  H  LOAD  1/10/91 


mClPANT  09  RUN  41,  16-20  MINS  H  LOAD  1/10/91 


RTICIPANT  09  RUN  41,  20-25  MINS  H  LOAD  1/10/91 


RTICIPANT  09  RUN  41,  25-30  MINS  H  LOAD  1/10/91 


C-39 


RTICIPANT  09  RUN  44,  0-5  MINS  L  LOAD  1/10/91 


tRTICIPANT  09  RUN  44,  ^0-15  MINS  L  LOAD  1/10/91 


PARTICIPANT  09  RUN  44,  15-20  MINS  L  LOAD  1/10/91 


C-41 
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